:W&Bá¾ÜèÆàM 1xyn37r0ª; 9 /0.29.0+7872fd9d9f86ca5c50617e19cb168122b7f959480.70.0~Ø ü‚" 7cs8zdimupnk 1xyn37r0ŠØ 1xyn37r0juju11-uc-san-diegoSPAtop"› _wandb‚{}  batch_size‚256  dataset_limit‚0.2 dropout‚0.036182545880756256 epochs‚10  hidden_dim‚64 ( learning_rate‚0.00010103477511329308  num_attention_heads‚8  num_branch_encoder_layers‚3  num_embedding_layers‚10  num_encoder_layers‚6Bjjtttt_vanilla_1xyn37r0b88oquqqwj)jj-spatop-sweepagent-tttt-vanilla-2-spfhcŠ ÜÜãÔ˜ÞñÚ3 (H¨¸hp"3.12.10*0.29.0j linux-x86_64Í…0$7 1xyn37r0&  _wandb.runtime‚0  _runtime‚0¹«!šjÒä -Linux-6.8.0-136-generic-x86_64-with-glibc2.36CPython 3.12.10 ÜÜãԈ߯ò*-of*=/data/spatop/options_files/sweep_tttt_jj/config_1xyn37r0.json*-l* /scratch/logs*-n*jjtttt_vanilla_1xyn37r02-m spanet.trainRjacksonjulian2014@gmail.comZ /scratch/logsb)jj-spatop-sweepagent-tttt-vanilla-2-spfhcr/opt/conda/bin/pythonº v6zaudyywsh32wb35d0wodqxkuc9ztmi$+œTÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€€Ã¯ü ª€àäÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-1512559d-7724-700d-39a3-dab660c9a52eÊ13.3º v6zaudyywsh32wb35d0wodqxkuc9ztmiµª½Ç2  requirements.txt‡º›W 1xyn37r0ZF (H¨¸ (H¨¸ (hpð"3.12.10*0.29.0j linux-x86_64ƒÑ^ïY 1xyn37r0ZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.29.0j linux-x86_64ì[[*  1xyn37r0b trainer/global_step2aÀ·t\  1xyn37r0ZK (H¨¸ (H¨¸(8hpð"3.12.10*0.29.0j linux-x86_64ÏPLó/  1xyn37r0b*"trainer/global_step2§K  1xyn37r0j: ÝÜãÔà„œÀ*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] gm5þ“  1xyn37r0j  ÝÜãÔðÿؤî â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 220 K │ train │ 0 │ │ 1 │ encoder │ JetEncoder │ 498 K │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 9.6 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 48 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ Ÿ¬Á\ 1xyn37r0jÊ ÝÜãÔ¸‰Ë¥·Trainable params: 10.3 M Non-trainable params: 86 Total params: 10.3 M Total estimated model params size (MB): 41.196 Modules in train mode: 4413 Modules in eval mode: 0 Total FLOPs: 0 Í¢`q 1xyn37r0j` ÝÜãÔÈŒä¥Pwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). 4ˆ{ 1xyn37r0jj ÝÜãÔÑï¥Zwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). ŒÛÊy 1xyn37r0jh ÝÜãÔàËö¥Xwandb: WARNING Config item 'num_encoder_layers' was locked by 'sweep' (ignored update). Ö^g€ 1xyn37r0jo ÝÜãÔȾý¥_wandb: WARNING Config item 'num_branch_encoder_layers' was locked by 'sweep' (ignored update). FÅ3…z 1xyn37r0ji ÝÜãÔƒ‰¦Ywandb: WARNING Config item 'num_attention_heads' was locked by 'sweep' (ignored update). ­\”t 1xyn37r0jc ÝÜãÔ¨à‘¦Swandb: WARNING Config item 'dataset_limit' was locked by 'sweep' (ignored update). æ˜Âq 1xyn37r0j` ÝÜãÔ •¦Pwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). 40)Yt 1xyn37r0jc ÝÜãÔ໥¦Swandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). "iï;n 1xyn37r0j] ÝÜãÔÈ¢°¦Mwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). s ™m 1xyn37r0j\ ÝÜãÔ€¨¶¦Lwandb: WARNING Config item 'epochs' was locked by 'sweep' (ignored update). ¸Íûm_ 1xyn37r0*Í  transformer_dim‚64  transformer_dim_scale‚2.0  initial_embedding_dim‚64  position_embedding_dim‚64 ! num_branch_embedding_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚0  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚""  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 F event_info_file‚2"/data/spatop/event_files/tttt/tttt_hadronic.yaml" = training_file‚+"/data/spatop/tttt_15M/h5/tttt_training.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"s£ë 1xyn37r0jÙ ÝÜãÔ å«È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. çÙ^@N 1xyn37r0j¼ ÝÜãÔð©««/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. ÚÎ<ë 1xyn37r0jÙ àÜãÔØýè‡È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ÿ\{P 1xyn37r0j¾ àÜãÔè܈­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. ¯±¥œ‚(:’ ìÜãÔÑ‘™ gpu.0.gpu‚36.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚30.0 gpu.0.powerWatts‚108.678' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.735599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾ ˜â‚(:Ø ìÜãÔõù»¢ disk.dm-0.out‚11.96337890625 disk.dm-1.out‚ 0.61328125 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd1777.in‚0# disk.nvme0n1.out‚11.96337890625 disk.rbd2.in‚0 disk.rbd3.out‚0% disk.nvme0c0n1.out‚11.96337890625& proc.memory.availableMB‚ 700.703125 cpu‚25.694355865156794 disk.dm-0.in‚0 disk.sda.out‚ 0.61328125 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd1.out‚0 disk.sda3.out‚ 0.61328125 disk.sda1.out‚0 network.sent‚34484) disk./.usagePercent‚70.80817922772071% disk./.usageGB‚1055.0930404663086 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0 disk.sda3.in‚0 disk.rbd2.out‚ 1.71484375 disk.rbd3.in‚0$ memory_percent‚95.72324752807617 proc.cpu.threads‚17 disk.nvme0n1.in‚0 disk.drbd1777.out‚0) proc.memory.percent‚16.98288917541504 network.recv‚39298 disk.dm-1.in‚0 disk.sda.in‚0" proc.memory.rssMB‚ 2782.4765625±W¨; ‚(:† ûÜãÔÁÌ®™ gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.102' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6204 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žþÛä!‚(:Ú ûÜãÔÆåœ disk.sda1.out‚0 disk.dm-1.out‚ 1.55859375 disk.nvme0c0n1.in‚0$ memory_percent‚95.72622776031494% disk./.usageGB‚1055.0930786132812 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.sda3.in‚0 network.sent‚44844 disk.rbd1.out‚0 disk.dm-0.out‚13.56982421875 disk.sda2.in‚0 disk.nvme0n1.in‚0# disk.nvme0n1.out‚13.56982421875) proc.memory.percent‚16.98298454284668 cpu‚25.518450079654464 disk.rbd2.in‚0( proc.memory.availableMB‚ 700.21484375" proc.memory.rssMB‚ 2782.4921875 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd2.out‚ 1.71484375% disk.nvme0c0n1.out‚13.56982421875 disk.sda.out‚ 1.55859375) disk./.usagePercent‚70.80818178779606 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 network.recv‚44252 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 1.55859375 disk.sda.in‚0—;Äa3"b/ lr-AdamW/pg1"trainer/global_step2:JP<Ñsˆ3#b/ lr-AdamW/pg2"trainer/global_step2:JPK-Jò-$) _wandbruntime‚37  _runtime‚37¥Cx¿%º _step‚0 & lr-AdamW/pg1‚4.094875087304682e-06 & lr-AdamW/pg2‚4.094875087304682e-06 trainer/global_step‚49 % _timestamp‚1.7884074269556189e+09 _runtime‚ 37.36782306²µMÁ&¼ trainer/global_step‚49 % _timestamp‚1.7884074269556189e+09 _runtime‚ 37.36782306 _step‚0 & lr-AdamW/pg1‚4.094875087304682e-06 & lr-AdamW/pg2‚4.094875087304682e-06ÙðЛx'bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP¶«Nþf(bb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JP%Hè2j)bf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JP ¦…ìh*bd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JPÞóŠxi+be BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JP#4COn,bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JP((µiv-br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JP³ËˆL_.b[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JPH…Ö‹p/bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JP¿ÿåd0b` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JP+ªk1bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JP4N0ªs2bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JP®À—o3bk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JP£Š¨Ji4be BDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak"trainer/global_step2:JPù[Ïs5bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPÜŽ3k6bg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JPÉ©n7bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPÃ^Å:h8bd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JP‰Xr9bn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JPd 8k:bg DDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated"trainer/global_step2:JP/Ü^Ïj;bf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JPaZ˜jbc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JPhJ¨o?bk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JPS½©n@bj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JP™ úœ`Ab\ 9DeviceStatsMonitor.on_train_batch_start/segment.all.freed"trainer/global_step2:JP þCkBbg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JP¶\ÂïkCbg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JPv¾æ\gDbc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JPœ?–jEbf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JP´;ñ3iFbe BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JPÒÀ¶fGbb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JP-o^„hHbd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JP¿ª}nIbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPgÈqhJbd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JPAfc³iKbe BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JPH #¾mLbi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPŽÃcMb_ DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPÿì:Ùsbbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPÞÔ÷bfcbb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JPÛvAjdbf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JPŽ@GQgebc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JP‡ì?mfbi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JPNË!vcgb_ Ïvjbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JP£;iS]kbY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JPbäNï^lbZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JPÌF&Ajmbf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JPÄ&yrnbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPu^Ïv`ob\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JPÜUþŽppbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JP!݇Ódqb` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JPß‘Ct^rbZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JPY³¼esba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JP?üi³ctb_ DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JPãú'ƒf}bb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JPûn‰i~be BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JPâÂ_fbb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JPžúÝq€bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JP1æõšhbc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JP9£ ˜q‚bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JPèoÁ}oƒbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JPÇK”m„bh EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current"trainer/global_step2:JPatb…b] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JPÿÎul†bg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JP‰¯ñm‡bh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JPøωjˆbe BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JPº§¨®n‰bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JPw[æoŠbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JP®wc‹b^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JP!¾•ósŒbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JP=Nblbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JPx¾ÒŽtŽbo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP4¢|Wfba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPŒ!g™pbk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JP°|0o‘bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JP¹ È©t’bo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JP`ßdr“bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JP׿ýr”bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JPvqoc•b^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JP-¡)Šm–bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JP7Ä¥Ou—bp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPŸ/ËCm˜bh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JPã¡GÎi™bd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JPôCy>fšba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPó¨°”r›bm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JP·Urœbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JP[3&Ž`b[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JP±¿„úhžbc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JP?R§óoŸbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPäTHXk bf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JP”öär¡bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JP‡Uk.¢) _wandbruntime‚37  _runtime‚37'{+£üN XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 68936930304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22628 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚377054 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 1445050371072 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚ 37.368901912 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚652865 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚641717 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚986703 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 68737321486 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚641715 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 1445236606976 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 1527093168640 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 1307036081152 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚986703 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237701 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22628 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚641715 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 1376131443712 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911252992 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 69105163264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 1445236606976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_tra^JÌ^W in_batch_start/segment.all.current‚416 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚377056 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 1445050371072 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 243680256 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 68902435342 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22901 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 1376113440768 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 70256657920 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 1375938516494 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 1527336848896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 70476398080 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 1375756363278 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚275279 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911252992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409944064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 1376131443712ypœÌ†'¤€O SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 1375756363278 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚275279 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911252992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409944064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 1376131443712 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 68936930304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22628 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚377054 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 1445050371072 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚ 37.368901912 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚652865 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚641717 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚986703 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 68737321486 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚641715 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 1445236606976 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 1527093168640 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 1307036081152 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚986703 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237701 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22628 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚641715 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 1376131443712 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911252992 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 69105163264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 1445236606976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚377056 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 1445050371072 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 243680256 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 68902435342 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22901 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 1376113440768 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 70256657920 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 1375938516494 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 1527336848896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 70476398080 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216qѯk¥bf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JP蹯o¦bj GDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current"trainer/global_step2:JP'©©rd§b_ gÊbb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JP¹Qy@hËbc @DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated"trainer/global_step2:JP8ÑznÌbi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JP ùB[lÍbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JPÎÍ=—dÎb_ !I0wÑbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JP‚"‚uÒbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JPo‰2çfÓba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JPõh(eÔb` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JP¡dA­fÕba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JPýfÀbÖb] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JPŠ-àjp×bk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JP4³…´]ØbX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JPT$bKfÙba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JP ŒeÚb` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JP%°ÁoÛbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JP ÈýÓgÜbb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JPÔe }rÝbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JP@øjÞbe BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JPOocµeßb` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JP1ĵlàbg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JPú·‹Úgább ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JPºöŒ_âbZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JPg^× hãbc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JP‚Jmäbh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JPûRŽnåbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JP”Àbfæba >DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JPéOpçbk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JPG|inèbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JP²ü×hébc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JP%ηjêbe BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JP0MÛfëba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JPu¾°‚pìbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JP VØfíba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JPj‰8hiîbd ADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated"trainer/global_step2:JPã¯ÒGiïbd ADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current"trainer/global_step2:JP'ˆlŸrðbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JPÈ•è£fñba >DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JPjáqòbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JP:¾ÈÚaób\ 9DeviceStatsMonitor.on_train_batch_end/segment.all.current"trainer/global_step2:JPÇ›Ôëiôbd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JPÆuÑõwõbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP÷‡oköbf CDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated"trainer/global_step2:JP…3•=e÷b` =DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak"trainer/global_step2:JPÙiaøb\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JP‰P»miùbd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JPb!H¸lúbg DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JP+}`ûb[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JPÉyôµ^übY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JPû¾ª—oýbj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JP­|7mþbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPåŒzÞgÿbb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JP³µŒÑj€be BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JP\AŽkbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JP§¯â¤r‚bm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JP¿JUklƒbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JPY÷¿çk„bf CDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JPI)¦Ão…bj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JPõ/= l†bg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JPŽ™bc‡b^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JP…S̱iˆbd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JPÐøfg‰bb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JP.fŠmŠbh EDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated"trainer/global_step2:JPÞîÙÓj‹be BDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JP$nœmŒbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JP“Ìldb_ DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPZ…öm•bh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPÙþ’^h–bc @DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak"trainer/global_step2:JP»º]j—be BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JPiguoq˜bl IDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JPƘÜOm™bh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JP}húâmšbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JPYÄZœp›bk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JPÞ(Ûsœbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP]z§Ó\bW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JP|óMdžb_ DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚665582 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚384433 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚280628 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 1557656169472 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 1485749458944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚49 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 71675626496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚654374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 1403400808448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚654376 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 70498314752 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1660864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1679432 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 1473899123200 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231084032 RDDeviceStatsMonitor.on_train_bAüvsÉatch_end/reserved_bytes.small_pool.peak‚ 417333248 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1006490 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 _step‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1679432 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚968 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 70291600094 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 1403088710158 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚ 38.015075347 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1025056 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 70330068480 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 1332962224128 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 1332979263488 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚665059 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚281149 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1006490 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 71882770432 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 1403270863582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1025056 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1557425085440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1660864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 1403400808448 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚384431 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0í}©¼’&¢ŒM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚654376 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 70498314752 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1660864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1679432 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 1473899123200 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231084032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1006490 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 _step‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1679432 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚968 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 70291600094 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 1403088710158 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚ 38.015075347 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1025056 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 70330068480 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 1332962224128 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 1332979263488 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚665059 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚281149 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1006490 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 71882770432 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 1403270863582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1025056 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1557425085440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1660864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 1403400808448 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚384431 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 1403382805504 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 70498314752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚947 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 1473899123200 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 1473712873984 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚654376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚665582 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚384433 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚280628 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 1557656169472 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 1485749458944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚49 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 71675626496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚654374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 1403400808448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2îjÈA£b< loss/FBt4/assignment_loss"trainer/global_step2:JPFÑ·)-¤b( epoch"trainer/global_step2:JPxcjpB¥b= loss/SRqqt3/detection_loss"trainer/global_step2:JPöiÞŠ@¦b; loss/FRt4/detection_loss"trainer/global_step2:JP’˜oB§b= loss/SRqqt1/detection_loss"trainer/global_step2:JP\’U@¨b; loss/FBt3/detection_loss"trainer/global_step2:JPç¿àÀ@©b; loss/FBt1/detection_loss"trainer/global_step2:JP™žî@ªb; loss/FBt4/detection_loss"trainer/global_step2:JPññÛîA«b< loss/FRt2/assignment_loss"trainer/global_step2:JP loss/SRqqt3/assignment_loss"trainer/global_step2:JPÍçk¹A±b< loss/FBt3/assignment_loss"trainer/global_step2:JPӕŘ@²b; loss/FBt2/detection_loss"trainer/global_step2:JPc¯A³b< loss/FBt1/assignment_loss"trainer/global_step2:JPÈ™¦C´b> loss/SRqqt1/assignment_loss"trainer/global_step2:JP}†Xå@µb; loss/FRt2/detection_loss"trainer/global_step2:JP¼G¾þ7¶b2 loss/total_loss"trainer/global_step2:JP#=ÓB·b= loss/SRqqt4/detection_loss"trainer/global_step2:JPãl«¾A¸b< loss/FRt3/assignment_loss"trainer/global_step2:JPƒD[fC¹b> loss/SRqqt2/assignment_loss"trainer/global_step2:JP¿ /@ºb; loss/FRt1/detection_loss"trainer/global_step2:JP¾)–'A»b< loss/FRt1/assignment_loss"trainer/global_step2:JP!}C¼b> loss/SRqqt4/assignment_loss"trainer/global_step2:JPWƒŽŠ.½) _wandbruntime‚38  _runtime‚38:PŠH¾ 0loss/SRqqt3/detection_loss‚2.598556984716387 trainer/global_step‚49 % _timestamp‚1.7884074276043665e+09 _runtime‚ 38.016453453 .loss/FRt4/detection_loss‚4.767235928318566 0loss/SRqqt1/detection_loss‚3.290562424570956 .loss/FBt3/detection_loss‚8.424682106484067 .loss/FBt1/detection_loss‚8.160419207811355 .loss/FBt4/detection_loss‚6.715203087776899 /loss/FRt2/assignment_loss‚7.332162857055664 /loss/FRt4/assignment_loss‚7.226379169358148 0loss/FBt2/assignment_loss‚2.7946508475712366 /loss/FRt3/detection_loss‚3.6720119749611997 0loss/SRqqt2/detection_loss‚3.643943951211192 1loss/SRqqt3/assignment_loss‚5.402310136538833 /loss/FBt3/assignment_loss‚2.801017024300315 .loss/FBt2/detection_loss‚5.064149198361806 _step‚3 /loss/FBt1/assignment_loss‚2.740201508998871 1loss/SRqqt1/assignment_loss‚5.517418242515402 .loss/FRt2/detection_loss‚3.789860484274951 &loss/total_loss‚120.37575006644988 0loss/SRqqt4/detection_loss‚4.142133374766606 /loss/FRt3/assignment_loss‚7.334412077639965 1loss/SRqqt2/assignment_loss‚5.440809791738337 /loss/FRt1/detection_loss‚3.9534652537625767 0loss/FRt1/assignment_loss‚7.4023193483767304 2loss/SRqqt4/assignment_loss‚5.4039117766589655 0loss/FBt4/assignment_loss‚2.7579333086808524 epoch‚0;ZºÔL¿Æ 0loss/SRqqt3/detection_loss‚2.598556984716387 trainer/global_step‚49 % _timestamp‚1.7884074276043665e+09 _runtime‚ 38.016453453 .loss/FRt4/detection_loss‚4.767235928318566 0loss/SRqqt1/detection_loss‚3.290562424570956 .loss/FBt3/detection_loss‚8.424682106484067 .loss/FBt1/detection_loss‚8.160419207811355 .loss/FBt4/detection_loss‚6.715203087776899 /loss/FRt2/assignment_loss‚7.332162857055664 /loss/FRt4/assignment_loss‚7.226379169358148 0loss/FBt2/assignment_loss‚2.7946508475712366 /loss/FRt3/detection_loss‚3.6720119749611997 0loss/SRqqt2/detection_loss‚3.643943951211192 1loss/SRqqt3/assignment_loss‚5.402310136538833 /loss/FBt3/assignment_loss‚2.801017024300315 .loss/FBt2/detection_loss‚5.064149198361806 _step‚3 /loss/FBt1/assignment_loss‚2.740201508998871 1loss/SRqqt1/assignment_loss‚5.517418242515402 .loss/FRt2/detection_loss‚3.789860484274951 &loss/total_loss‚120.37575006644988 0loss/SRqqt4/detection_loss‚4.142133374766606 /loss/FRt3/assignment_loss‚7.334412077639965 1loss/SRqqt2/assignment_loss‚5.440809791738337 /loss/FRt1/detection_loss‚3.9534652537625767 0loss/FRt1/assignment_loss‚7.4023193483767304 2loss/SRqqt4/assignment_loss‚5.4039117766589655 0loss/FBt4/assignment_loss‚2.7579333086808524 epoch‚0QRô-À‚(:„ ŠÝãÔФ·™ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.51' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.302 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qõ$ÎåÁ‚(:Ú ŠÝãÔ‘ò¤ disk.nvme0n1.in‚0% disk.nvme0c0n1.out‚13.90576171875 cpu‚25.41715975820158 network.sent‚71880 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 2.36328125 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 proc.memory.rssMB‚ 2782.53125 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 699.63671875) proc.memory.percent‚16.98322296142578 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda3.in‚0$ memory_percent‚95.72975635528564 network.recv‚79677) disk./.usagePercent‚70.80818562790905% disk./.usageGB‚1055.0931358337402 disk.sda1.in‚0 disk.dm-0.in‚0 disk.sda3.out‚ 2.36328125# disk.nvme0n1.out‚13.90576171875 disk.nvme0c0n1.in‚0 disk.sda.in‚0 disk.sda.out‚ 2.36328125 disk.dm-0.out‚13.90576171875 disk.rbd2.out‚ 1.81689453125ç¥ Â‚(:„ ™ÝãÔñ¨®™ gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.24' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.648 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽÒŽäÂ(:Ù ™ÝãÔ÷’Ø› disk.rbd1.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2782.62109375 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚0( proc.memory.availableMB‚ 699.00390625 network.sent‚135745 disk.sda2.out‚0 disk.sda.out‚3.09375 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0$ memory_percent‚95.73361873626709 cpu‚25.353068851302105( disk./.usagePercent‚70.8165703866705 disk.rbd2.out‚ 1.81689453125 disk.rbd3.out‚0 disk.sda1.out‚0* proc.memory.percent‚16.983771324157715$ disk./.usageGB‚1055.218074798584 disk.rbd0.out‚0 disk.rbd1.out‚0! disk.dm-0.out‚113.36474609375 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚0 disk.dm-1.out‚3.09375 network.recv‚111849 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚113.36474609375 disk.sda3.in‚0 disk.sda3.out‚3.09375$ disk.nvme0n1.out‚113.36474609375 disk.drbd1777.out‚0ùúB”.Ä) _wandbruntime‚71  _runtime‚71?t—ÁÁÅ» & lr-AdamW/pg2‚8.273319053942114e-06 trainer/global_step‚99 % _timestamp‚1.7884074613271132e+09 _runtime‚ 71.739250591 _step‚4 & lr-AdamW/pg1‚8.273319053942114e-06!¥<ÐÅÆ¿ % _timestamp‚1.7884074613271132e+09 _runtime‚ 71.739250591 _step‚4 & lr-AdamW/pg1‚8.273319053942114e-06 & lr-AdamW/pg2‚8.273319053942114e-06 trainer/global_step‚99l„‰«.Ç) _wandbruntime‚71  _runtime‚71ÒM—‘'È‹O XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3043613899776 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚954 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 _step‚5 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 2739581677568 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3043847094272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _runtime‚ 71.740262547 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 2739599680512 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 2878189930496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚556 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 138776485888 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 2604195197952 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 2878376166400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚549628 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1274667 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 2604178158592 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1997945 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 2739599680512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚549072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 138776485888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 2742569451702 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1979405 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 138608252928 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1294976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚99 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 $ _timestamp‚1.788407461327822e+09 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1274667 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1295534 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1274665 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 2739581677568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1979405 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 2902507862016 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 138374253750 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3254070 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 2878376166400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1274665 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 260159488 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 138209139894 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3272612 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1997945 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2878189930496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚745906 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 141339232256 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 141129977856 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 2742387298486 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚558 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209254400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 138776485888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 2742569451702 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1979405 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 138608252928 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1294976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚99 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 $ _timestamp‚1.788407461327822e+09 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1274667 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1295534 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1274665 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 2739581677568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1979405 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 2902507862016 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 138374253750 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3254070 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 2878376166400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1274665 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 260159488 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 138209139894 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3272612 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1997945 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2878189930496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚745906 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 141339232256 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 141129977856 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 2742387298486 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚558 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209254400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 235278336 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚99 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2017758 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3286516 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1307948 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2017758 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 2630138380288 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1287326 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 2907038682624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 139598304438 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1999192 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3305084 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚975 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 3073944831488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 72.386752763 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 2907038682624 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1287324 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚530 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 2630121340928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 139763418502 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚753283 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 2931420810240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0µ56 &ÌšM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1308478 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 142547961344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1307948 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2017758 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 2630138380288 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1287326 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 2907038682624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 139598304438 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1999192 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3305084 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚975 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 3073944831488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 72.386752763 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 2907038682624 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1287324 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚530 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 2630121340928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 139763418502 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚753283 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 2931420810240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 2769901798790 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 140169637376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 2766869045248 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1999192 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚554667 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 3074180109824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 140001391104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 235278336 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚99 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2017758 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3286516 FëÕ‚(:à ÆÝãÔµñîœ cpu‚25.247383803262103 network.sent‚199430 disk.dm-0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0* proc.memory.percent‚16.983914375305176 disk.sda.in‚0# disk.nvme0n1.out‚213.1376953125 disk.dm-0.out‚213.1376953125% disk.nvme0c0n1.out‚213.1376953125 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 3.59423828125 disk.rbd3.in‚0 network.recv‚128532 disk.dm-1.out‚ 4.8984375 disk.rbd3.out‚0$ memory_percent‚95.73593139648438) disk./.usagePercent‚70.82497844211755$ disk./.usageGB‚1055.343360900879 disk.sda.out‚ 4.8984375 proc.cpu.threads‚17 disk.nvme0n1.in‚0 disk.nvme0c0n1.in‚0 disk.sda3.out‚ 4.8984375# proc.memory.rssMB‚ 2782.64453125 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.rbd1.out‚ 0.2109375# proc.memory.availableMB‚698.625+±âÁ0Ö+ _wandbruntime‚106  _runtime‚106®bÂyÅ׿ _step‚8 ' lr-AdamW/pg1‚1.2451763020579545e-05 ' lr-AdamW/pg2‚1.2451763020579545e-05 trainer/global_step‚149 % _timestamp‚1.7884074956952825e+09 _runtime‚ 106.107410214ë!XXÉØà ' lr-AdamW/pg1‚1.2451763020579545e-05 ' lr-AdamW/pg2‚1.2451763020579545e-05 trainer/global_step‚149 % _timestamp‚1.7884074956952825e+09 _runtime‚ 106.107410214 _step‚8ûÐÖù0Ù+ _wandbruntime‚106  _runtime‚106Ù|ˆ–'ÚO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 4348131333120 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 4311515725824 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1907617 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _step‚9 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1907617 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 4311329489920 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚826049 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3901354314752 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4311329489920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 4109200386910 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2990647 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1940803 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1114754 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 260159488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 207157248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 208447808512 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 4103067917312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 4103049914368 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4560336301568 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 208279575552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 231097344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 207680958302 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1907615 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 % _timestamp‚1.7884074956960545e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1114756 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4879722 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1907615 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2972107 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4898264 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4348155273216 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 211973871104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 4311515725824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O Óš'Û”O WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1114756 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4879722 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1907615 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2972107 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4898264 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4348155273216 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 211973871104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 4311515725824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 4348131333120 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 4311515725824 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1907617 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _step‚9 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1907617 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 4311329489920 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚826049 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3901354314752 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4311329489920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 4109200386910 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2990647 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1940803 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1114754 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 260159488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 207157248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 208447808512 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 4103067917312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 4103049914368 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4560336301568 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 208279575552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 231097344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 207680958302 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1907615 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 % _timestamp‚1.7884074956960545e+09 Kð:î0Ü+ _wandbruntime‚106  _runtime‚106ŠËGƒÅ Ý™M B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3010460 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1953784 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 4130337282048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 4130319279104 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 209070122846 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 $ _timestamp‚1.788407496342597e+09 _runtime‚ 106.75525144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 209672713728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 trainer/global_step‚149 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4930736 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3927280457728 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 4340178242048 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 213393052672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 209840960000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4377068221440 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4912168 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2991894 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2991894 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 209235236910 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1954340 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3010460 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1920274 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231084032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚832207 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 209840960000 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1920276 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 4136532733998 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4930736 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 4136350580574 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _step‚10 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚988 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4912168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1122131 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚831653 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4590668418048 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 433999199283¢OŒ.Ú2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚967 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4590437334016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4377044281344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 260159488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 209070122846 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 $ _timestamp‚1.788407496342597e+09 _runtime‚ 106.75525144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 209672713728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 trainer/global_step‚149 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4930736 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3927280457728 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 4340178242048 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 213393052672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 209840960000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4377068221440 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4912168 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2991894 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2991894 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 209235236910 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1954340 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3010460 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1920274 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231084032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚832207 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 209840960000 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1920276 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 4136532733998 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4930736 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 4136350580574 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _step‚10 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚988 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4912168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1122131 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚831653 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4590668418048 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 4339991992832 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚967 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4590437334016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4377044281344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 260159488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1953784 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 äí±U0ß+ _wandbruntime‚106  _runtime‚106ƒŠóDྠ/loss/FRt4/detection_loss‚3.8110220935098673 1loss/SRqqt1/detection_loss‚1.8159438832788855 .loss/FBt4/detection_loss‚8.590087137724224 /loss/FRt3/detection_loss‚2.6098364757709818 % _timestamp‚1.7884074963443506e+09 /loss/FRt1/detection_loss‚2.9293692787659578 /loss/FRt4/assignment_loss‚7.203576349076771 /loss/FBt2/assignment_loss‚2.747702678044637 .loss/FBt3/assignment_loss‚2.78439403463293 1loss/SRqqt4/assignment_loss‚5.408535518144307 /loss/SRqqt4/detection_loss‚4.41609847898546 ,loss/FBt2/detection_loss‚4.3689453907502 /loss/FRt1/assignment_loss‚7.423074870273985 epoch‚0 _step‚11 .loss/FBt3/detection_loss‚6.652941551473406 0loss/FBt4/assignment_loss‚2.7417958912096525 .loss/FRt3/assignment_loss‚7.18062539178817 1loss/SRqqt3/assignment_loss‚5.310780729566302 /loss/FBt1/assignment_loss‚2.677015595965915 _runtime‚ 106.756390676 /loss/FRt2/assignment_loss‚7.279661271307203 1loss/SRqqt1/assignment_loss‚5.533023318728885 0loss/SRqqt2/detection_loss‚4.956257607900735 .loss/FBt1/detection_loss‚18.59904771712091 %loss/total_loss‚128.1780770113203 trainer/global_step‚149 -loss/FRt2/detection_loss‚4.75223374615113 2loss/SRqqt2/assignment_loss‚5.4162795760414815 0loss/SRqqt3/detection_loss‚2.969828425108322}PoHá _step‚11 .loss/FBt3/detection_loss‚6.652941551473406 0loss/FBt4/assignment_loss‚2.7417958912096525 .loss/FRt3/assignment_loss‚7.18062539178817 1loss/SRqqt3/assignment_loss‚5.310780729566302 /loss/FBt1/assignment_loss‚2.677015595965915 _runtime‚ 106.756390676 /loss/FRt2/assignment_loss‚7.279661271307203 1loss/SRqqt1/assignment_loss‚5.533023318728885 0loss/SRqqt2/detection_loss‚4.956257607900735 .loss/FBt1/detection_loss‚18.59904771712091 %loss/total_loss‚128.1780770113203 trainer/global_step‚149 -loss/FRt2/detection_loss‚4.75223374615113 2loss/SRqqt2/assignment_loss‚5.4162795760414815 0loss/SRqqt3/detection_loss‚2.969828425108322 /loss/FRt4/detection_loss‚3.8110220935098673 1loss/SRqqt1/detection_loss‚1.8159438832788855 .loss/FBt4/detection_loss‚8.590087137724224 /loss/FRt3/detection_loss‚2.6098364757709818 % _timestamp‚1.7884074963443506e+09 /loss/FRt1/detection_loss‚2.9293692787659578 /loss/FRt4/assignment_loss‚7.203576349076771 /loss/FBt2/assignment_loss‚2.747702678044637 .loss/FBt3/assignment_loss‚2.78439403463293 1loss/SRqqt4/assignment_loss‚5.408535518144307 /loss/SRqqt4/detection_loss‚4.41609847898546 ,loss/FBt2/detection_loss‚4.3689453907502 /loss/FRt1/assignment_loss‚7.423074870273985 epoch‚0 \Q—â‚(:… ÕÝãÔ¢¹š™ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0﹕íã‚(:â ÕÝãÔøÐÁ› disk.rbd1.out‚ 0.2109375" proc.memory.rssMB‚ 2782.6484375 network.sent‚206035 disk.rbd0.in‚0# disk.nvme0n1.out‚312.8837890625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.rbd2.out‚ 3.60986328125% disk.nvme0c0n1.out‚312.8837890625 disk.sda2.in‚0 disk.dm-1.out‚ 5.8515625 disk.rbd0.out‚0) disk./.usagePercent‚70.84176690419797$ disk./.usageGB‚1055.593521118164 disk.rbd2.in‚0 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚0 disk.sda3.in‚0 disk.nvme0n1.in‚0 disk.dm-1.in‚0$ memory_percent‚95.74346542358398* proc.memory.percent‚16.983938217163086 cpu‚25.225671700355065 disk.drbd1777.in‚0 network.recv‚133394 disk.sda2.out‚0 disk.sda3.out‚ 5.8515625 disk.sda.out‚ 5.8515625 disk.sda1.out‚0& proc.memory.availableMB‚ 697.390625 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.dm-0.in‚0 disk.dm-0.out‚312.8837890625}ÞŸý‘ä‚(:† äÝãÔ툹™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.492' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6984 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|æ©Oìå‚(:á äÝãÔ¼üƒœ network.recv‚138703 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda1.in‚0) disk./.usagePercent‚70.84176255206988 disk.sda2.out‚0* proc.memory.percent‚16.983962059020996 network.sent‚255311% disk./.usageGB‚1055.5934562683105$ disk.nvme0c0n1.out‚ 313.263671875 disk.sda3.in‚0 disk.sda.out‚6.296875 cpu‚25.21055897208871 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.nvme0n1.in‚0" disk.nvme0n1.out‚ 313.263671875 disk.dm-1.out‚6.296875 disk.rbd2.out‚ 3.7626953125 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda3.out‚6.296875 disk.sda1.out‚0$ memory_percent‚95.74353694915771( proc.memory.availableMB‚ 697.37890625# proc.memory.rssMB‚ 2782.65234375 disk.rbd1.out‚ 0.2431640625 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.rbd1.in‚0 disk.dm-0.out‚ 313.263671875 disk.nvme0c0n1.in‚0 disk.rbd3.in‚0oʲü0æ+ _wandbruntime‚140  _runtime‚140¯PRÅç¿ _runtime‚ 140.400545853 _step‚12 ' lr-AdamW/pg1‚1.6630206987216976e-05 ' lr-AdamW/pg2‚1.6630206987216976e-05 trainer/global_step‚199 $ _timestamp‚1.788407529988357e+09­¢Éèà ' lr-AdamW/pg1‚1.6630206987216976e-05 ' lr-AdamW/pg2‚1.6630206987216976e-05 trainer/global_step‚199 $ _timestamp‚1.788407529988357e+09 _runtime‚ 140.400545853 _step‚12 ¿«†Ä0é+ _wandbruntime‚140  _runtime‚1401#™'ê“O G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 5793778744320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198768640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚515 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3983349 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _step‚13 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1483604 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2540567 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5744469049344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 5466518151168 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3983349 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 5744655285248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222708736 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884075299891276e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1103025 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6076812139008 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 283009454592 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 278119131136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _runtime‚ 140.401490611 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6505374 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3964809 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5198513431552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1483606 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2540565 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6523916 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2586631 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 5466536154112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 5475831322118 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2540567 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1102510 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2586114 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 278119131136 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6076589430272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5198496392192 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 277152776710 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3964809 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 282810685952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚517 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 5793802684416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 5466536154112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 5475831322118 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2540567 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1102510 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2586114 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 278119131136 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6076589430272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5198496392192 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 277152776710 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3964809 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 282810685952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚517 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 5793802684416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 5793778744320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198768640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversiñj°Àze_allocations.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚515 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3983349 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _step‚13 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1483604 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2540567 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5744469049344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 5466518151168 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3983349 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 5744655285248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222708736 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884075299891276e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1103025 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6076812139008 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 283009454592 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 278119131136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _runtime‚ 140.401490611 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6505374 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3964809 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5198513431552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1483606 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2540565 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6523916 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2586631 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 279512282624 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1490981 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884075306328404e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5493805518848 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5502981515782 _runtime‚ 141.045657865 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 5773317801472 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚495 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1108103 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6556388 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6537820 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4003162 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2599084 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2553226 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚989 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 278541941254 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233181184 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5503163669206 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1108598 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5822691692544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 279344036352 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6556388 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2599581 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 6107152130560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6537820 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2553224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 5773131552256 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2553226 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 6106918949376 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 278707055318 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3984596 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 5224439574528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 5493787515904 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 5773317801472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968•Ó§&î¡M H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 5773131552256 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2553226 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 6106918949376 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 278707055318 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3984596 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 5224439574528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 5493787515904 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 5773317801472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2553224 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4003162 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 284227256832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3984596 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 209241088 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 5493805518848 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 5224456613888 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 279512282624 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1490981 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884075306328404e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5493805518848 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5502981515782 _runtime‚ 141.045657865 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 5773317801472 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚495 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1108103 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6556388 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6537820 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4003162 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2599084 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2553226 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚989 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 278541941254 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233181184 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5503163669206 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1108598 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5822691692544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 279344036352 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6556388 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2599581 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 6107152130560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6537820 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2553224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3173517 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 6495672548352 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3173515 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 6842280104110 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3173517 ZHDeviceStatsMonitor.½«1Á!on_train_batch_start/allocated_bytes.large_pool.freed‚ 6829986387968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚972 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7177608608768 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3231525 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚513 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1378558 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7177608608768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 346624595118 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4976051 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4957511 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚511 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7177794844672 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4976051 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚993 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 7239426155520 _step‚17 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235291648 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 211351552 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 6495655508992 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 346789708974 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 347790453760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3231012 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 7593437931008 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3173515 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 7239450095616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1852456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 347790453760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 6830004390912 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 347622220800 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 6842462257326 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8131026 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8149568 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 7593202639360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884075642891243e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 353776483840 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 347622220800 trainer/global_step‚249XÿÀÊ'û—O ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 211351552 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 6495655508992 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 346789708974 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 347790453760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3231012 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 7593437931008 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3173515 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 7239450095616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1852456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 347790453760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 6830004390912 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 347622220800 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 6842462257326 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8131026 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8149568 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 7593202639360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884075642891243e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 353776483840 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 347622220800 trainer/global_step‚249 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 6830004390912 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1379069 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 6829986387968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1852454 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3173517 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 6495672548352 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3173515 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 6842280104110 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3173517 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 6829986387968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚972 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7177608608768 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3231525 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚513 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1378558 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7177608608768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 346624595118 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4976051 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4957511 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚511 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7177794844672 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4976051 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚993 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 7239426155520 _step‚17 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235291648 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408&E+80ü+ _wandbruntime‚175  _runtime‚175öM/q£&ýM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4995864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7206457360896 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1384081 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233181184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 6869612450990 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4977298 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884075649287024e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4977298 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 209241088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3186174 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8182040 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7206457360896 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 349015358976 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 6521598691328 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 6857273755648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚993 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8163472 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1859831 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚18 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4995864 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3186176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1384604 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 6857255752704 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 6521615730688 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8163472 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 7268363043840 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _runtime‚ 175.341474141 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3186176 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3244437 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 355198641664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 349015358976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 7623770926592 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7623537745408 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 7268339103744 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 7206271111680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 trainer/global_step‚249 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 349183605248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3243912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 348178873726 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚972 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 6869794604414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚06ˆ› mþ¡M UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7206457360896 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1384081 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233181184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 6869612450990 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4977298 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884075649287024e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4977298 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 209241088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3186174 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8182040 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7206457360896 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 349015358976 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 6521598691328 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 \JDeviwä ::ceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 6857273755648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚993 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8163472 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1859831 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚18 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4995864 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3186176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1384604 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 6857255752704 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 6521615730688 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8163472 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 7268363043840 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _runtime‚ 175.341474141 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3186176 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3244437 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 355198641664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 349015358976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 7623770926592 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7623537745408 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 7268339103744 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 7206271111680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 trainer/global_step‚249 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 349183605248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3243912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 348178873726 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚972 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 6869794604414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4995864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416‰X ƒ0ÿ+ _wandbruntime‚175  _runtime‚1750ì³^K€Å /loss/FRt3/assignment_loss‚7.230007928515238 0loss/SRqqt2/assignment_loss‚5.51132532119751 0loss/SRqqt1/detection_loss‚2.496365529452932 0loss/SRqqt2/detection_loss‚6.606327397823334 _step‚19 /loss/FRt1/assignment_loss‚7.371613897126297 /loss/FBt4/assignment_loss‚2.351021303070916 /loss/FRt1/detection_loss‚2.9636002642841173 /loss/FBt2/detection_loss‚1.4777261127243817 trainer/global_step‚249 /loss/FRt4/assignment_loss‚7.221352728930387 1loss/SRqqt1/assignment_loss‚5.512426738081308 0loss/FBt2/assignment_loss‚2.4304561376571656 0loss/SRqqt4/detection_loss‚3.646229325429253 .loss/FRt2/detection_loss‚6.090566953378064 .loss/FRt4/detection_loss‚5.925223354595705 .loss/FBt1/detection_loss‚14.87901530211622 .loss/FBt3/detection_loss‚9.104675456881523 &loss/total_loss‚127.75111598261913 /loss/FRt2/assignment_loss‚7.373474223273141 1loss/SRqqt3/detection_loss‚2.5756290695862845 epoch‚0 % _timestamp‚1.7884075649307678e+09 1loss/SRqqt4/assignment_loss‚5.403706643892371 /loss/FBt1/assignment_loss‚2.235656044699929 0loss/FBt3/assignment_loss‚2.4045138597488402 /loss/FRt3/detection_loss‚2.5444491083423295 .loss/FBt4/detection_loss‚9.055474291245142 _runtime‚ 175.342852327 2loss/SRqqt3/assignment_loss‚5.3402789905667305 X®±OÉ /loss/FRt3/assignment_loss‚7.230007928515238 0loss/SRqqt2/assignment_loss‚5.51132532119751 0loss/SRqqt1/detection_loss‚2.496365529452932 0loss/SRqqt2/detection_loss‚6.606327397823334 _step‚19 /loss/FRt1/assignment_loss‚7.371613897126297 /loss/FBt4/assignment_loss‚2.351021303070916 /loss/FRt1/detection_loss‚2.9636002642841173 /loss/FBt2/detection_loss‚1.4777261127243817 trainer/global_step‚249 /loss/FRt4/assignment_loss‚7.221352728930387 1loss/SRqqt1/assignment_loss‚5.512426738081308 0loss/FBt2/assignment_loss‚2.4304561376571656 0loss/SRqqt4/detection_loss‚3.646229325429253 .loss/FRt2/detection_loss‚6.090566953378064 .loss/FRt4/detection_loss‚5.925223354595705 .loss/FBt1/detection_loss‚14.87901530211622 .loss/FBt3/detection_loss‚9.104675456881523 &loss/total_loss‚127.75111598261913 /loss/FRt2/assignment_loss‚7.373474223273141 1loss/SRqqt3/detection_loss‚2.5756290695862845 epoch‚0 % _timestamp‚1.7884075649307678e+09 1loss/SRqqt4/assignment_loss‚5.403706643892371 /loss/FBt1/assignment_loss‚2.235656044699929 0loss/FBt3/assignment_loss‚2.4045138597488402 /loss/FRt3/detection_loss‚2.5444491083423295 .loss/FBt4/detection_loss‚9.055474291245142 _runtime‚ 175.342852327 2loss/SRqqt3/assignment_loss‚5.34027899056673058øU‚‚(:… ‘Þãԃ㙠gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚92.829' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5658 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uþ<žñƒ‚(:æ ‘ÞãÔӻ㜠disk.nvme0n1.in‚0 disk.dm-1.out‚8.296875 disk.rbd0.in‚0 network.recv‚154549 disk.sda3.out‚8.296875 disk.sda.in‚0! disk.dm-0.out‚512.43017578125 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.30615234375$ disk.nvme0n1.out‚512.43017578125 disk.sda3.in‚0 disk.rbd3.in‚0 disk.drbd1777.in‚0 proc.memory.rssMB‚ 2782.65625% disk./.usageGB‚1055.5936431884766 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd2.in‚0 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚512.43017578125 disk.sda1.out‚0* proc.memory.percent‚16.983985900878906) disk./.usagePercent‚70.84177509643906 cpu‚25.173839218795344 proc.cpu.threads‚17 disk.dm-0.in‚0 network.sent‚318111 disk.rbd2.out‚ 5.4306640625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚8.296875$ memory_percent‚96.62425518035889( proc.memory.availableMB‚ 553.08203125(œ“®š„‚(:  ÞãÔâ’¢š gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.47' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.093999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ä|ó…‚(:è  ÞãÔöšð› disk.sda3.in‚0 proc.memory.rssMB‚ 2782.65625 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 5.50146484375$ disk.nvme0n1.out‚611.60986328125& disk.nvme0c0n1.out‚611.60986328125 disk.sda.out‚ 8.9453125* proc.memory.percent‚16.983985900878906 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 552.72265625 proc.cpu.threads‚17) disk./.usagePercent‚70.87533435608002 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 8.9453125 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.dm-1.out‚ 8.9453125 disk.rbd0.in‚0 network.recv‚160181% disk./.usageGB‚1056.0937004089355 disk.rbd1.in‚0 disk.nvme0n1.in‚0 disk.nvme0c0n1.in‚0$ memory_percent‚96.62644863128662 cpu‚25.165594632138767 disk.rbd1.out‚ 0.369140625! disk.dm-0.out‚611.60986328125 disk.sda.in‚0 network.sent‚367378 disk.dm-0.in‚0 disk.dm-1.in‚0à±0†+ _wandbruntime‚209  _runtime‚209˜åCć¾ _step‚20 & lr-AdamW/pg1‚2.498709492049184e-05 & lr-AdamW/pg2‚2.498709492049184e-05 trainer/global_step‚299 % _timestamp‚1.7884075986244686e+09 _runtime‚ 209.036631229J ÅȈ & lr-AdamW/pg1‚2.498709492049184e-05 & lr-AdamW/pg2‚2.498709492049184e-05 trainer/global_step‚299 % _timestamp‚1.7884075986244686e+09 _runtime‚ 209.036631229 _step‚20ÌñÔ‰0‰+ _wandbruntime‚209  _runtime‚209TYðm˜'Š’O VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8610748168192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2221306 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2221304 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3806465 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 8193472627712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 8193472627712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8685073566720 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 8208911039318 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7792831665152 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 8193454624768 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 417461776384 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 417293543424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3806467 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3875892 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 424828588032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202962944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9756678 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 trainer/global_step‚299 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1654068 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 424625625088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 417293543424 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8610748168192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7792814625792 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 416096413526 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 416261527382 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8610934404096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944Fì>€œ'‹–O KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8610934404096 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5968753 _step‚21 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2221306 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2221304 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3806465 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 8193472627712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 8193472627712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8685073566720 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 8208911039318 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7792831665152 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 8193454624768 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 417461776384 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 417293543424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3806467 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3875892 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 424828588032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202962944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9756678 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 trainer/global_step‚299 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1654068 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 424625625088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 417293543424 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8610748168192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7792814625792 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 416096413526 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 416261527382 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8610934404096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8610748168192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3888171 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 _runtime‚ 209.680438247 _step‚22 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3819124 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9789124 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 418854927872 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3819126 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 426045918720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 8639596920320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3819124 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2228681 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 8220723989504 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9789124 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7818757808128 % _timestamp‚1.7884075992679017e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9807692 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 8220741992448 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5988566 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3819126 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 8220741992448 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 8236425539622 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5970000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1659490 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 NDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9140259323392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9807692 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 426248868352 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 418854927872 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3888680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚509 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 8236243386198 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚972 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 8639410671104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚299 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 418686681600 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9140032433664 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5988566 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 418686681600 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚993 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 417485578070 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2228683 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9140259323392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9807692 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 426248868352 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 418854927872 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3888680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚509 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 8236243386198 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚972 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 8639410671104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚299 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 418686681600 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9140032433664 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5988566 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 418686681600 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚993 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 417485578070 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2228683 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3888171 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 _runtime‚ 209.680438247 _step‚22 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3819124 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9789124 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 418854927872 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3819126 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 426045918720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 8639596920320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3819124 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2228681 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 8220723989504 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9789124 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7818757808128 % _timestamp‚1.7884075992679017e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9807692 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 8220741992448 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5988566 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3819126 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 8220741992448 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 8236425539622 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5970000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1659490 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199Ó̳%0+ _wandbruntime‚209  _runtime‚209DILÆ .loss/FRt2/detection_loss‚4.119045687157933 /loss/FRt1/assignment_loss‚7.399056066546524 /loss/FRt3/assignment_loss‚7.234691373763546 0loss/SRqqt2/detection_loss‚7.132813532715258 epoch‚0 % _timestamp‚1.7884075992695398e+09 _runtime‚ 209.681577283 /loss/FRt1/detection_loss‚3.0101153102883123 &loss/total_loss‚141.59308163167157 trainer/global_step‚299 _step‚23 /loss/FRt2/assignment_loss‚7.279931568517918 1loss/SRqqt3/assignment_loss‚5.288717940233756 1loss/SRqqt4/assignment_loss‚5.332499112838354 0loss/FBt3/assignment_loss‚1.8597133484753696 /loss/FBt2/detection_loss‚1.1616746027777682 .loss/FBt3/detection_loss‚8.239278995177962 1loss/SRqqt1/assignment_loss‚5.427647911134313 /loss/FRt4/detection_loss‚5.5265922885070955 1loss/SRqqt3/detection_loss‚2.3747623527179593 1loss/SRqqt2/assignment_loss‚5.423149647920028 0loss/FBt4/assignment_loss‚1.6777488191922505 0loss/SRqqt4/detection_loss‚4.267220087158374 0loss/FBt1/assignment_loss‚1.5791412711143493 1loss/SRqqt1/detection_loss‚2.2294159994506444 -loss/FRt3/detection_loss‚2.57547302916646 -loss/FBt4/detection_loss‚27.3845058431228 /loss/FRt4/assignment_loss‚7.095700740814209 /loss/FBt2/assignment_loss‚1.726067332757844 /loss/FBt1/detection_loss‚16.248118770122527­“²P‘Ê 0loss/FBt4/assignment_loss‚1.6777488191922505 0loss/SRqqt4/detection_loss‚4.267220087158374 0loss/FBt1/assignment_loss‚1.5791412711143493 1loss/SRqqt1/detection_loss‚2.2294159994506444 -loss/FRt3/detection_loss‚2.57547302916646 -loss/FBt4/detection_loss‚27.3845058431228 /loss/FRt4/assignment_loss‚7.095700740814209 /loss/FBt2/assignment_loss‚1.726067332757844 /loss/FBt1/detection_loss‚16.248118770122527 .loss/FRt2/detection_loss‚4.119045687157933 /loss/FRt1/assignment_loss‚7.399056066546524 /loss/FRt3/assignment_loss‚7.234691373763546 0loss/SRqqt2/detection_loss‚7.132813532715258 epoch‚0 % _timestamp‚1.7884075992695398e+09 _runtime‚ 209.681577283 /loss/FRt1/detection_loss‚3.0101153102883123 &loss/total_loss‚141.59308163167157 trainer/global_step‚299 _step‚23 /loss/FRt2/assignment_loss‚7.279931568517918 1loss/SRqqt3/assignment_loss‚5.288717940233756 1loss/SRqqt4/assignment_loss‚5.332499112838354 0loss/FBt3/assignment_loss‚1.8597133484753696 /loss/FBt2/detection_loss‚1.1616746027777682 .loss/FBt3/detection_loss‚8.239278995177962 1loss/SRqqt1/assignment_loss‚5.427647911134313 /loss/FRt4/detection_loss‚5.5265922885070955 1loss/SRqqt3/detection_loss‚2.3747623527179593 1loss/SRqqt2/assignment_loss‚5.423149647920028˜ÖJœ’‚(:‘ ¯ÞãÔ˜µ´™ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚120.123' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.024600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*Ãö“‚(:ë ¯ÞãÔå×Ï›% disk./.usageGB‚1056.0937004089355 network.recv‚163718 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 0.369140625 disk.nvme0n1.in‚0! disk.dm-0.out‚612.10986328125 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 9.3828125& disk.nvme0c0n1.out‚612.10986328125 disk.sda1.in‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚612.10986328125 network.sent‚373143 disk.dm-1.in‚0 disk.dm-1.out‚ 9.3828125 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda.out‚ 9.3828125 disk.drbd1777.in‚0 disk.drbd1777.out‚0) disk./.usagePercent‚70.87533435608002( proc.memory.availableMB‚ 553.19921875 proc.cpu.threads‚17 disk.dm-0.in‚0 disk.sda3.in‚0$ memory_percent‚96.62353992462158* proc.memory.percent‚16.984009742736816 cpu‚25.158260587402243 disk.rbd0.in‚0 disk.rbd2.out‚ 5.51708984375 disk.rbd3.out‚0 disk.nvme0c0n1.in‚0# proc.memory.rssMB‚ 2782.66015625 F]j‘”‚(:† ¾ÞãÔ“ôº™ gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚104.207' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.8414 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0} EEë•‚(:à ¾ÞãÔý¬Í›% disk./.usageGB‚1056.0937042236328 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 5.51708984375 disk.sda3.out‚ 9.80078125) disk./.usagePercent‚70.87533461208754 disk.sda.in‚0 disk.nvme0n1.in‚0 disk.dm-1.out‚ 9.80078125 disk.rbd3.in‚0 disk.sda2.in‚0# memory_percent‚96.6243028640747# proc.memory.rssMB‚ 2782.66015625 network.sent‚422817 disk.sda3.in‚0 network.recv‚170583 disk.nvme0n1.out‚ 711.3046875 disk.nvme0c0n1.in‚0 disk.sda.out‚ 9.80078125 disk.rbd1.in‚0 disk.dm-0.in‚0 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0* proc.memory.percent‚16.984009742736816 cpu‚25.15075734649997 disk.drbd1777.in‚0 disk.dm-0.out‚ 711.3046875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 0.369140625 disk.rbd0.out‚0" disk.nvme0c0n1.out‚ 711.3046875( proc.memory.availableMB‚ 553.07421875nþè‘–‚(:† ÍÞãÔ¼–ˆ™ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.983' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9966 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»2BWì—‚(:á ÍÞãÔ••Âœ" proc.memory.rssMB‚ 2782.6640625% disk./.usageGB‚1056.0937042236328 disk.sda1.out‚0! disk.nvme0n1.out‚ 811.99609375 disk.rbd3.in‚0# disk.nvme0c0n1.out‚ 811.99609375 disk.sda2.in‚0 disk.sda3.out‚ 10.5234375 cpu‚25.144867802159823 network.sent‚429357 disk.sda3.in‚0 disk.dm-0.out‚ 811.99609375 disk.dm-1.out‚ 10.5234375 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.dm-1.in‚0$ memory_percent‚96.62613868713379) disk./.usagePercent‚70.87533461208754 disk.sda2.out‚0 disk.rbd1.out‚ 0.369140625 disk.nvme0n1.in‚0 disk.dm-0.in‚0* proc.memory.percent‚16.984033584594727 disk.sda.in‚0 disk.drbd1777.in‚0 proc.cpu.threads‚17 network.recv‚175356 disk.rbd2.in‚0 disk.rbd2.out‚ 5.685546875 disk.nvme0c0n1.in‚0 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 10.5234375 disk.rbd1.in‚0' proc.memory.availableMB‚ 552.7734375‹Év0˜+ _wandbruntime‚243  _runtime‚243ÈpÄ™¾ _step‚24 & lr-AdamW/pg1‚2.916553888712927e-05 & lr-AdamW/pg2‚2.916553888712927e-05 trainer/global_step‚349 % _timestamp‚1.7884076326484425e+09 _runtime‚ 243.060546401j¼Ï¤ÈšÂ & lr-AdamW/pg1‚2.916553888712927e-05 & lr-AdamW/pg2‚2.916553888712927e-05 trainer/global_step‚349 % _timestamp‚1.7884076326484425e+09 _runtime‚ 243.060546401 _step‚24$’BÍ0›+ _wandbruntime‚243  _runtime‚243€©ì1œžO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚10043887727616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚972 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚10130744918016 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6942915 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚10043887727616 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4518245 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 % _timestamp‚1.7884076326492152e+09 _step‚25 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2590156 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 9575541974526 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 9556940864512 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4439415 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚10044073963520 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 211351552 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6942915 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 9575724127742 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 486964866048 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 495551457792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 trainer/global_step‚349 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚993 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4518789 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6961455 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 9089990781952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11382330 _runtime‚ 243.06178803 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4439417 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4439417 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 9089973742592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 9556940864512 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 485733345790 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚10044073963520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235291648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 9556922861568 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 486964866048¶õÎI¨'¢O VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 486964866048 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 495551457792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 trainer/global_step‚349 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚993 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4518789 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6961455 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 9089990781952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11382330 _runtime‚ 243.06178803 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4439417 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4439417 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 9089973742592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 9556940864512 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 485733345790 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚10044073963520 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235291648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 9556922861568 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 486964866048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚10043887727616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚972 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚10130744918016 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6942915 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚10043887727616 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4518245 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 % _timestamp‚1.7884076326492152e+09 _step‚25 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2590156 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 9575541974526 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 9556940864512 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4439415 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚10044073963520 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 211351552 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6942915 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 9575724127742 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421”²®Ú0ž+ _wandbruntime‚243  _runtime‚243ÊMq®&Ÿ¨M ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6981268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10159657866240 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 496966574080 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11414776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 _step‚26 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11433344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 487122510542 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2597533 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1933441 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 486957396478 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 9584192226304 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4530972 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 239472640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚10072550230528 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11414776 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 488358004224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 9602874321406 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788407633284434e+09 _runtime‚ 243.696983697 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6981268 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4452074 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 9115916924928 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4452074 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 9584210229248 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 9603056474830 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 trainer/global_step‚349 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4531480 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 9584192226304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 488526250496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1933947 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2597531 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚506 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 497182106624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚10072736479744 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 488526250496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4452076 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160ÿ;_‘ä  ¬M PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 488358004224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 9602874321406 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788407633284434e+09 _runtime‚ 243.696983697 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6981268 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4452074 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 9115916924928 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4452074 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 9584210229248 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 9603056474830 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 trainer/global_step‚349 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4531480 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 9584192226304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 488526250496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1933947 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2597531 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚506 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 497182106624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚10072736479744 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 488526250496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4452076 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6981268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10159657866240 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 496966574080 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11414776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 _step‚26 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11433344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 487122510542 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2597533 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1933441 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 486957396478 UCDeviceStatsMonitor.on_train_batch_end/acto:6Îive_bytes.large_pool.freed‚ 9584192226304 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4530972 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 239472640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚10072550230528 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11414776 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416{(ô0¡+ _wandbruntime‚243  _runtime‚243ùVÛžM¢Ç .loss/FRt2/detection_loss‚4.720327898859978 %loss/total_loss‚132.5738601308008 epoch‚0 0loss/SRqqt2/detection_loss‚8.418376102259284 /loss/FRt2/assignment_loss‚7.373843524191114 0loss/FBt2/assignment_loss‚0.7766756225715984 .loss/FRt1/detection_loss‚3.752693801470425 /loss/FRt3/detection_loss‚2.3124657023757225 /loss/FBt2/detection_loss‚1.2475112697668946 1loss/SRqqt1/assignment_loss‚5.051818798570072 0loss/SRqqt4/assignment_loss‚5.02453438802199 0loss/FBt3/assignment_loss‚1.2827398509814822 0loss/FBt4/assignment_loss‚1.0216314991315205 0loss/SRqqt3/detection_loss‚2.696304328739643 1loss/SRqqt4/detection_loss‚3.7537817866964773 /loss/FBt1/detection_loss‚26.768381903568905 % _timestamp‚1.7884076332859924e+09 1loss/SRqqt2/assignment_loss‚5.146116708454333 /loss/FBt3/detection_loss‚5.9753519846447585 trainer/global_step‚349 .loss/FRt1/assignment_loss‚7.44283594255862 /loss/FRt4/assignment_loss‚7.276901192135281 1loss/SRqqt1/detection_loss‚1.7192022135987988 2loss/SRqqt3/assignment_loss‚4.9610483841817885 0loss/FBt1/assignment_loss‚1.2874896327654521 _runtime‚ 243.698039416 _step‚27 .loss/FRt3/assignment_loss‚7.14525342366052 .loss/FRt4/detection_loss‚6.697967006100574 /loss/FBt4/detection_loss‚10.720607165495554ñŠb>Q£Ë 0loss/SRqqt3/detection_loss‚2.696304328739643 1loss/SRqqt4/detection_loss‚3.7537817866964773 /loss/FBt1/detection_loss‚26.768381903568905 % _timestamp‚1.7884076332859924e+09 1loss/SRqqt2/assignment_loss‚5.146116708454333 /loss/FBt3/detection_loss‚5.9753519846447585 trainer/global_step‚349 .loss/FRt1/assignment_loss‚7.44283594255862 /loss/FRt4/assignment_loss‚7.276901192135281 1loss/SRqqt1/detection_loss‚1.7192022135987988 2loss/SRqqt3/assignment_loss‚4.9610483841817885 0loss/FBt1/assignment_loss‚1.2874896327654521 _runtime‚ 243.698039416 _step‚27 .loss/FRt3/assignment_loss‚7.14525342366052 .loss/FRt4/detection_loss‚6.697967006100574 /loss/FBt4/detection_loss‚10.720607165495554 .loss/FRt2/detection_loss‚4.720327898859978 %loss/total_loss‚132.5738601308008 epoch‚0 0loss/SRqqt2/detection_loss‚8.418376102259284 /loss/FRt2/assignment_loss‚7.373843524191114 0loss/FBt2/assignment_loss‚0.7766756225715984 .loss/FRt1/detection_loss‚3.752693801470425 /loss/FRt3/detection_loss‚2.3124657023757225 /loss/FBt2/detection_loss‚1.2475112697668946 1loss/SRqqt1/assignment_loss‚5.051818798570072 0loss/SRqqt4/assignment_loss‚5.02453438802199 0loss/FBt3/assignment_loss‚1.2827398509814822 0loss/FBt4/assignment_loss‚1.0216314991315205† 󑤂(:† ÜÞãÔç…¶™ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.064' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8128 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kMpëꥂ(:ß ÜÞãÔª»œ disk.sda.out‚ 11.2890625 disk.nvme0n1.in‚0% disk./.usageGB‚1056.0937614440918" disk.nvme0c0n1.out‚ 812.3828125 disk.rbd3.in‚0 disk.dm-0.in‚0 disk.sda3.out‚ 11.2890625 disk.drbd1777.in‚0$ memory_percent‚96.62497043609619 network.sent‚435399 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 0.369140625 disk.nvme0n1.out‚ 812.3828125# proc.memory.rssMB‚ 2782.66796875 cpu‚25.13978721201832) disk./.usagePercent‚70.87533845220057 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1777.out‚0( proc.memory.availableMB‚ 552.96484375 network.recv‚179818 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.sda1.out‚0* proc.memory.percent‚16.984057426452637 disk.rbd3.out‚0 disk.sda.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.rbd2.out‚ 7.259765625 disk.dm-0.out‚ 812.3828125 disk.dm-1.out‚ 11.2890625âeÚ‘¦‚(:† ëÞãÔÀñ®™ gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚123.758' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7516 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹U§‚(:ý ëÞãÔÐéØœ disk.sda3.out‚ 11.98828125 disk.sda.in‚0 disk.sda.out‚ 11.98828125 disk.sda1.out‚0 disk.dm-0.out‚ 912.029296875 disk.rbd3.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚70.87534408436632 disk.sda1.in‚0" disk.nvme0n1.out‚ 912.029296875 disk.rbd2.out‚ 7.337890625 cpu‚25.134327887963675 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.0078125 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 0.0078125# proc.memory.rssMB‚ 2782.66796875 disk.drbd1777.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda2.out‚0# memory_percent‚96.6238021850586% proc.memory.availableMB‚ 553.15625 network.recv‚184942 disk.dm-1.in‚0 disk.dm-1.out‚ 11.98828125$ disk.nvme0c0n1.out‚ 912.029296875 network.sent‚484812% disk./.usageGB‚1056.0938453674316 disk.rbd1.in‚0 disk.rbd1.out‚ 0.369140625 disk.dm-0.in‚ 0.0078125 disk.rbd2.in‚0 disk.sda2.in‚0* proc.memory.percent‚16.984057426452637F£&0¨+ _wandbruntime‚277  _runtime‚277cTö:©¼ % lr-AdamW/pg2‚3.33439828537667e-05 trainer/global_step‚399 % _timestamp‚1.7884076668986883e+09 _runtime‚ 277.310857899 _step‚28 % lr-AdamW/pg1‚3.33439828537667e-05ØûÆYƪÀ % lr-AdamW/pg2‚3.33439828537667e-05 trainer/global_step‚399 % _timestamp‚1.7884076668986883e+09 _runtime‚ 277.310857899 _step‚28 % lr-AdamW/pg1‚3.33439828537667e-05é¾­Ä0«+ _wandbruntime‚277  _runtime‚277ìµ£f¬'¬¦O MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚10920391098368 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 556804421632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚507 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 556636188672 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚10920391098368 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 555040050342 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _step‚29 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚10942355062950 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7954157 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11477027287040 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7954157 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2204648 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚11477213522944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5072365 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚10920409101312 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 556636188672 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2204141 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚993 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚12142840649216 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚10920409101312 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5072365 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 556804421632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _runtime‚ 277.311875095 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7935617 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5072367 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2959004 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚11477027287040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚10942172909734 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 trainer/global_step‚399 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7935617 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5163145 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13026524 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12143069649408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11576368389120 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚10387149898752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 $ _timestamp‚1.788407666899446e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚509 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 566677320192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 555205164198 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 566472260096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚10387132859392 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚11477213522944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960œÆ ;°'­ªO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5163654 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚972 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5072367 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚11576392329216 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 205060096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚10920409101312 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5072365 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 556804421632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _runtime‚ 277.311875095 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7935617 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5072367 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2959004 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚11477027287040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚10942172909734 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 trainer/global_step‚399 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7935617 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5163145 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13026524 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12143069649408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11576368389120 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚10387149898752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 $ _timestamp‚1.788407666899446e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚509 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 566677320192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 555205164198 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 566472260096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚10387132859392 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚11477213522944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚10920391098368 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 556804421632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚507 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 556636188672 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚10920391098368 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 555040050342 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _step‚29 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚10942355062950 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7954157 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11477027287040 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7954157 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2204648 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚11477213522944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5072365 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚10920409101312 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 556636188672 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2204141 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚993 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚12142840649216 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 % _timestamp‚1.7884076675417368e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 558029326848 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12173401396224 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5175811 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚10413093081088 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 556429214886 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 558197573120 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 558197573120 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7955404 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚972 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7973970 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚10947678466048 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 556594328950 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 277.954488123 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2966381 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11605281337344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚496 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚11505876039168 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13040428 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7973970 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 567888974848 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚10947660463104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13040428 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5085024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13058996Y”»&°µM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 % _timestamp‚1.7884076675417368e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 558029326848 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12173401396224 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5175811 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚10413093081088 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 556429214886 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 558197573120 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 558197573120 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7955404 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚972 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7973970 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚10947678466048 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 556594328950 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 277.954488123 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2966381 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11605281337344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚496 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚11505876039168 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13040428 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7973970 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 567888974848 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚10947660463104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13040428 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5085024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13058996 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5085024 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 211351552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚12308803844942 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚12910353082368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11684309015552 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5705315 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3327854 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14633634 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8928319 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2477751 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚12283877338112 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 626475744256 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5705317 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8928319 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235291648 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚12910166846464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3327856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5705317 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5806155 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8946859 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚12283877338112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚12308985998158 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 624676982606 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 637301555712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 626307511296 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5805605 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5705315 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 626475744256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚993 $ _timestamp‚1.788407701059468e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12910166846464 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8946859 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚13022039740416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2478299 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _runtime‚ 311.471888918 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚12910353082368 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚12283859335168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 624511868750 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14652176 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚12283859335168 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚972 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚449 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 626307511296 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13022015800320 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚11684291976192 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 637512907264e¿n½ªO $ _timestamp‚1.788407701059468e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12910166846464 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8946859 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚13022039740416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2478299 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _runtime‚ 311.471888918 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceSt*ñç®"atsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚12910353082368 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚12283859335168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 624511868750 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14652176 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚12283859335168 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚972 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚449 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 626307511296 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13022015800320 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚11684291976192 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 637512907264 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 211351552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚12308803844942 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚12910353082368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11684309015552 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5705315 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3327854 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14633634 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8928319 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2477751 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚12283877338112 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 626475744256 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5705317 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8928319 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235291648 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚12910166846464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3327856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5705317 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5806155 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8946859 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚12283877338112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚12308985998158 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 624676982606 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 637301555712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 626307511296 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5805605 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5705315 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 626475744256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚993!ƒåð0¾+ _wandbruntime‚312  _runtime‚312¥P¶·&¿±M C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14666080 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 638717542400 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 625901033294 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚13050952688640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14684648 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 627868895744 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚12336318345246 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚12311128699904 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2483876 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚12336136191822 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚12311128699904 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5818590 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚13689883666432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚12939015598592 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚12939015598592 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚12311146702848 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5819109 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 627700649472 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13689646290944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚449 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8966672 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14684648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5717976 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 627868895744 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _step‚34 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 627700649472 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8966672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7884077016982384e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5717974 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14666080 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5717976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2483359 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3335233 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 237375488 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8948106 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11710252197888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚13050952688640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14684648 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 627868895744 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚12336318345246 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚12311128699904 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2483876 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚12336136191822 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚12311128699904 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5818590 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚13689883666432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚12939015598592 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚12939015598592 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚12311146702848 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5819109 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 627700649472 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13689646290944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚449 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8966672 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14684648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5717976 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 627868895744 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _step‚34 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 627700649472 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8966672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7884077016982384e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5717974 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14666080 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5717976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2483359 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3335233 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 237375488 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8948106 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11710252197888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14343306405888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6338265 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2751488 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 213448704 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9921021 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3696706 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚499 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 694148801014 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 695978833920 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6338267 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 708271862784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13647345574912 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚13647345574912 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚12981451092992 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 $ _timestamp‚1.788407735558618e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚13647327571968 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚14343492641792 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6448736 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚13675434780150 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 _step‚37 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6338265 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚15175935074304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚12981468132352 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚ 345.970959012 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 696147066880 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14343492641792 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9921021 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚14467663211520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14343306405888 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3696704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 693983687158 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚972 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14343306405888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6338265 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2751488 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 213448704 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9921021 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3696706 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚499 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 694148801014 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 695978833920 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6338267 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 708271862784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13647345574912 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚13647345574912 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚12981451092992 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 $ _timestamp‚1.788407735558618e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚13647327571968 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚14343492641792 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6448736 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚13675434780150 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 _step‚37 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6338265 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚15175935074304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚12981468132352 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚ 345.970959012 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 696147066880 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14343492641792 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9921021 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚14467663211520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14343306405888 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3696704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 693983687158 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚972 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14372155158016 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚14372155158016 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚13674614939648 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14496576159744 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13674596936704 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3704083 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚972 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚38 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 709899171328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15206499271168 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚499 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9959374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9959374 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚528 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16291732 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚530 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚13674614939648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 709692027392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231084032 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚13702949280454 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚ 346.630800315 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14371968908800 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16310300 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16291732 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6461832 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 697540218368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 697540218368 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6461302 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15206268187136 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9940808 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 695537965766 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 695372851702 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 697371972096 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6350924 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14496600099840 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16310300 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6350924 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚137é6œ·02767127030 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 % _timestamp‚1.7884077362182415e+09 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9940808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚13007411314688 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14372155158016 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚14372155158016 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚13674614939648 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14496576159744 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13674596936704 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3704083 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚972 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚38 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 709899171328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15206499271168 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚499 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9959374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9959374 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚528 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16291732 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚530 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚13674614939648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 709692027392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231084032 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚13702949280454 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚ 346.630800315 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14371968908800 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16310300 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16291732 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6461832 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 697540218368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 697540218368 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6461302 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15206268187136 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9940808 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 695537965766 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 695372851702 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 697371972096 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6350924 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14496600099840 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16310300 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6350924 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚13702767127030 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 % _timestamp‚1.7884077362182415e+09 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9940808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚13007411314688 F0Ý+ _wandbruntime‚380  _runtime‚380ƒ"Ãѱ'Þ«O E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17884938 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 229000192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 763620619422 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7090021 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚16692646311424 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10932263 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3024467 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15042065715358 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10913723 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚547 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 779311748608 trainer/global_step‚549 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10932263 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7090568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚15776632201216 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 765650156544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚15010813811712 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4065556 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚972 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚15913334562816 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 763455505566 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 765650156544 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 765818389504 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6971217 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14278627249152 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚41 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 765818389504 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 779106688512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚ 380.369225142 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15913310622720 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 205060096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚15776445965312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚545 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚16692417311232 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚15010813811712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15776445965312 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6971215 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚993 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14278610209792 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚15776632201216}qy߯O IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10913723 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚547 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 779311748608 trainer/global_step‚549 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10932263 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7090568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚15776632201216 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 765650156544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚15010813811712 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4065556 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚972 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚15913334562816 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 763455505566 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 765650156544 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 765818389504 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6971217 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14278627249152 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚41 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 765818389504 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 779106688512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 O@D„§5Š<eviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚ 380.369225142 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15913310622720 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 205060096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚15776445965312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚545 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚16692417311232 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚15010813811712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15776445965312 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6971215 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚993 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14278610209792 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚15776632201216 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17884938 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 229000192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 763620619422 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7090021 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚16692646311424 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10932263 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3024467 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15042065715358)Ù{:0à+ _wandbruntime‚381  _runtime‚381Ž!l»&áµM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6983876 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15805108468224 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10952076 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚541 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚15069580215662 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14304553392128 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 764844670110 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10952076 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚15038065173504 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 767043294720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚16722745863680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15069398062238 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 767211540992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 780731533824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15805294717440 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15942247511040 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _step‚42 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10933510 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7103462 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 780522292736 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 trainer/global_step‚549 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15942223570944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4072933 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6983874 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14304570431488 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚15805294717440 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 767043294720 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 767211540992 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10933510 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17935952 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6983876 % _timestamp‚1.7884077705950208e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17935952 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7102919 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233181184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 765009784174 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚15038065173504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _runtime‚ 381.007527098¥C^¿&â¹M KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17935952 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6983876 % _timestamp‚1.7884077705950208e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17935952 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7102919 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233181184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 765009784174 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚15038065173504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _runtime‚ 381.007527098 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6983876 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15805108468224 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10952076 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚541 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚15069580215662 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14304553392128 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 764844670110 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10952076 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚15038065173504 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 767043294720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚16722745863680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15069398062238 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 767211540992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 780731533824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15805294717440 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15942247511040 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _step‚42 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10933510 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7103462 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 780522292736 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 trainer/global_step‚549 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15942223570944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4072933 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6983874 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14304570431488 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚15805294717440 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 767043294720 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 767211540992 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10933510 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0*vf±¬0ã+ _wandbruntime‚381  _runtime‚381Î÷b™PäÊ /loss/FRt1/detection_loss‚3.3631814475737367 2loss/SRqqt1/assignment_loss‚3.6504520558296365 _runtime‚ 381.00856356 1loss/SRqqt2/detection_loss‚6.7187102648119135 1loss/SRqqt4/detection_loss‚3.6793501561676916 .loss/FBt2/detection_loss‚3.690702442525561 trainer/global_step‚549 $ _timestamp‚1.788407770596526e+09 1loss/FBt3/assignment_loss‚0.24831076183131542 /loss/FRt2/detection_loss‚3.8599941726951372 /loss/FRt4/assignment_loss‚7.158245966984675 epoch‚0 1loss/SRqqt2/assignment_loss‚3.680435578028361 /loss/FBt3/detection_loss‚2.2004900415074724 %loss/total_loss‚97.16103154629926 _step‚43 0loss/FBt2/assignment_loss‚0.6103515566580675 -loss/FRt3/detection_loss‚2.58117393353733 /loss/FRt4/detection_loss‚4.4381973674664135 1loss/SRqqt3/detection_loss‚1.6298894733896083 .loss/FBt4/detection_loss‚10.58801374832789 0loss/FBt4/assignment_loss‚0.5990604300051927 0loss/FRt2/assignment_loss‚7.3204075154804045 /loss/FRt3/assignment_loss‚7.166286023996644 /loss/FRt1/assignment_loss‚7.422019360112209 1loss/SRqqt1/detection_loss‚2.8259839287690323 1loss/SRqqt3/assignment_loss‚3.328706235596628 1loss/SRqqt4/assignment_loss‚3.481013921174136 0loss/FBt1/assignment_loss‚0.4095041607118522 .loss/FBt1/detection_loss‚6.510551003118356;ÖÈTåÎ 1loss/SRqqt3/detection_loss‚1.6298894733896083 .loss/FBt4/detection_loss‚10.58801374832789 0loss/FBt4/assignment_loss‚0.5990604300051927 0loss/FRt2/assignment_loss‚7.3204075154804045 /loss/FRt3/assignment_loss‚7.166286023996644 /loss/FRt1/assignment_loss‚7.422019360112209 1loss/SRqqt1/detection_loss‚2.8259839287690323 1loss/SRqqt3/assignment_loss‚3.328706235596628 1loss/SRqqt4/assignment_loss‚3.481013921174136 0loss/FBt1/assignment_loss‚0.4095041607118522 .loss/FBt1/detection_loss‚6.510551003118356 /loss/FRt1/detection_loss‚3.3631814475737367 2loss/SRqqt1/assignment_loss‚3.6504520558296365 _runtime‚ 381.00856356 1loss/SRqqt2/detection_loss‚6.7187102648119135 1loss/SRqqt4/detection_loss‚3.6793501561676916 .loss/FBt2/detection_loss‚3.690702442525561 trainer/global_step‚549 $ _timestamp‚1.788407770596526e+09 1loss/FBt3/assignment_loss‚0.24831076183131542 /loss/FRt2/detection_loss‚3.8599941726951372 /loss/FRt4/assignment_loss‚7.158245966984675 epoch‚0 1loss/SRqqt2/assignment_loss‚3.680435578028361 /loss/FBt3/detection_loss‚2.2004900415074724 %loss/total_loss‚97.16103154629926 _step‚43 0loss/FBt2/assignment_loss‚0.6103515566580675 -loss/FRt3/detection_loss‚2.58117393353733 /loss/FRt4/detection_loss‚4.4381973674664135+±šæ‚(: ãßãÔÞ±¨™ gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.58' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.715999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/üvVç‚(:… ãßãÔ¬·˜œ disk.nvme0c0n1.in‚ 0.0234375* proc.memory.percent‚16.984105110168457 network.sent‚665153 disk.sda.in‚0 disk.drbd1777.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd2.out‚ 9.5185546875' disk.nvme0c0n1.out‚1411.43994140625 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.10877835540519 disk.sda1.out‚0% disk.nvme0n1.out‚1411.43994140625 disk.rbd3.out‚0# proc.memory.rssMB‚ 2782.67578125 disk.rbd1.out‚ 0.369140625" disk.dm-0.out‚1411.43994140625 disk.sda3.in‚0$ memory_percent‚96.62556648254395 network.recv‚224618 disk.nvme0n1.in‚ 0.0234375' proc.memory.availableMB‚ 552.8671875$ disk./.usageGB‚1056.058853149414 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 0.0234375 disk.dm-1.out‚ 17.0078125 disk.rbd0.out‚0 disk.sda3.out‚ 17.0078125 proc.cpu.threads‚17) disk./.usagePercent‚70.87299572725473 disk.sda.out‚ 17.0078125 disk.rbd2.in‚0 disk.rbd3.in‚0¯:››è‚(: òßãÔôš¯™ gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚89.829' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.965799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sf n é‚(:þ òßãÔûìЛ) disk./.usagePercent‚70.87301467181227 disk.sda3.in‚0 disk.sda.out‚ 17.58984375 disk.drbd1777.out‚0 disk.sda1.in‚0 disk.drbd1777.in‚0$ memory_percent‚96.62513732910156* proc.memory.percent‚16.984105110168457% disk./.usageGB‚1056.0591354370117 disk.dm-1.out‚ 17.58984375 disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚0.03125 disk.sda2.out‚0 disk.nvme0n1.in‚0.03125$ proc.memory.availableMB‚552.9375# proc.memory.rssMB‚ 2782.67578125 network.sent‚715093 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 9.5185546875 disk.sda.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.369140625! disk.dm-0.out‚1411.8486328125 disk.rbd3.in‚0 disk.sda3.out‚ 17.58984375 cpu‚25.106212538107396 network.recv‚230518 disk.dm-0.in‚0.03125 disk.rbd0.in‚0& disk.nvme0c0n1.out‚1411.8486328125 disk.sda2.in‚0$ disk.nvme0n1.out‚1411.8486328125ì…Ô30ê+ _wandbruntime‚414  _runtime‚414ç«ÊÑÆëÀ trainer/global_step‚599 % _timestamp‚1.7884078041868005e+09 _runtime‚ 414.598954516 _step‚44 ' lr-AdamW/pg1‚5.0057758720316424e-05 ' lr-AdamW/pg2‚5.0057758720316424e-05žÐ¾ßÊìÄ trainer/global_step‚599 % _timestamp‚1.7884078041868005e+09 _runtime‚ 414.598954516 _step‚44 ' lr-AdamW/pg1‚5.0057758720316424e-05 ' lr-AdamW/pg2‚5.0057758720316424e-05,–³Êö0í+ _wandbruntime‚414  _runtime‚414™"aÖ˜î«O UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11924965 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚993 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚16374282048512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 835321479168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11906425 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7604167 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚17358958033920 WJDeviceStatsMonitor.on_trainl“È!_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884078041875415e+09 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4434404 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚17209585524736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 835321479168 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 835489712128 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚ 414.599957124 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 835489712128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚16374282048512 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15575769326592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11906425 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11924965 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚972 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚17358981974016 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 trainer/global_step‚599 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 _step‚45 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7604167 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3298694 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 849962217472 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 832927323974 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7733098 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 850171471872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209254400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17209585524736 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 833092437830 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚16374264045568 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3299225 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚15575786365952 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚17209771760640 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚17209771760640 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233194496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19510590 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚18209153445888 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚18208920251392 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7604165 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540E¨Ãˆµ'ï¯O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚ 414.599957124 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 835489712128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚16374282048512 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15575769326592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11906425 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11924965 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚972 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚17358981974016 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 trainer/global_step‚599 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 _step‚45 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7604167 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3298694 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 849962217472 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 832927323974 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7733098 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 850171471872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209254400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17209585524736 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 833092437830 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚16374264045568 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3299225 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚15575786365952 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚17209771760640 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚17209771760640 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233194496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19510590 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚18209153445888 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚18208920251392 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7604165 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11924965 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚993 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚16374282048512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 835321479168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11906425 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7604167 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚17358958033920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884078041875415e+09 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4434404 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚17209585524736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 835321479168 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 835489712128 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11926212 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11944778 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16401533410304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7745855 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚993 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16436211150870 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15601712508928 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚599 _runtime‚ 415.240873502 _step‚46 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11944778 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚15601729548288 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 836714617344 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7616826 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚17387870982144 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 851371451904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7616826 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 836714617344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16401551413248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚17387894922240 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3304074 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 215532544 % _timestamp‚1.7884078048282843e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3304621 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 836882863616 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11926212 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7746404 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4441781 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚972 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19561604 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 239472640¤ØõÖ$ò¹M J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11926212 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11944778 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16401533410304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7745855 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚993 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16436211150870 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15601712508928 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚599 _runtime‚ 415.240873502 _step‚46 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11944778 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚15601729548288 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 836714617344 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7616826 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚17387870982144 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 851371451904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7616MœÇ›826 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 836714617344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16401551413248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚17387894922240 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3304074 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 215532544 % _timestamp‚1.7884078048282843e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3304621 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 836882863616 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11926212 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7746404 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4441781 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚972 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19561604 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 239472640 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚16401551413248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7616824 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424.øÀG&0ó+ _wandbruntime‚415  _runtime‚415Ú|q³QôË 1loss/SRqqt3/assignment_loss‚3.287473034858704 1loss/FBt1/assignment_loss‚0.38841119670094204 0loss/FBt2/assignment_loss‚0.6069873164479549 1loss/SRqqt1/detection_loss‚2.1070115952510777 .loss/FBt1/detection_loss‚5.913612351967738 .loss/FBt4/detection_loss‚9.069863405297784 /loss/FRt1/assignment_loss‚7.464906863677196 1loss/SRqqt3/detection_loss‚2.2721658157635467 /loss/FBt3/detection_loss‚1.3871212864294649 _runtime‚ 415.242105423 0loss/FRt3/assignment_loss‚7.1413501219315965 /loss/FRt2/detection_loss‚3.1945012777435537 0loss/SRqqt2/detection_loss‚5.406778948060397 $ _timestamp‚1.788407804830057e+09 .loss/FRt1/detection_loss‚4.408539714339452 0loss/FRt2/assignment_loss‚7.3883711756492145 0loss/FBt4/assignment_loss‚0.7490295760114404 .loss/FRt4/detection_loss‚4.486648970416614 0loss/FBt3/assignment_loss‚0.3430379576777341 %loss/total_loss‚91.10749101766855 epoch‚0 1loss/SRqqt1/assignment_loss‚3.291373954878913 1loss/SRqqt4/assignment_loss‚3.426493822444569 /loss/FRt3/detection_loss‚2.6243104236369783 _step‚47 /loss/FRt4/assignment_loss‚7.324353984424046 2loss/SRqqt2/assignment_loss‚3.5403106459255875 1loss/SRqqt4/detection_loss‚2.7564585384997455 .loss/FBt2/detection_loss‚2.528379039634307 trainer/global_step‚599«ñ`ïUõÏ /loss/FRt1/assignment_loss‚7.464906863677196 1loss/SRqqt3/detection_loss‚2.2721658157635467 /loss/FBt3/detection_loss‚1.3871212864294649 _runtime‚ 415.242105423 0loss/FRt3/assignment_loss‚7.1413501219315965 /loss/FRt2/detection_loss‚3.1945012777435537 0loss/SRqqt2/detection_loss‚5.406778948060397 $ _timestamp‚1.788407804830057e+09 .loss/FRt1/detection_loss‚4.408539714339452 0loss/FRt2/assignment_loss‚7.3883711756492145 0loss/FBt4/assignment_loss‚0.7490295760114404 .loss/FRt4/detection_loss‚4.486648970416614 0loss/FBt3/assignment_loss‚0.3430379576777341 %loss/total_loss‚91.10749101766855 epoch‚0 1loss/SRqqt1/assignment_loss‚3.291373954878913 1loss/SRqqt4/assignment_loss‚3.426493822444569 /loss/FRt3/detection_loss‚2.6243104236369783 _step‚47 /loss/FRt4/assignment_loss‚7.324353984424046 2loss/SRqqt2/assignment_loss‚3.5403106459255875 1loss/SRqqt4/detection_loss‚2.7564585384997455 .loss/FBt2/detection_loss‚2.528379039634307 trainer/global_step‚599 1loss/SRqqt3/assignment_loss‚3.287473034858704 1loss/FBt1/assignment_loss‚0.38841119670094204 0loss/FBt2/assignment_loss‚0.6069873164479549 1loss/SRqqt1/detection_loss‚2.1070115952510777 .loss/FBt1/detection_loss‚5.913612351967738 .loss/FBt4/detection_loss‚9.069863405297784/£eS’ö‚(:… àãÔçí©™ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.177' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6354 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Uê§= ÷‚(:ÿ àãÔ—÷²œ disk.dm-1.out‚ 18.2109375" proc.memory.rssMB‚ 2782.6796875 disk.nvme0n1.in‚0.03125 disk.rbd0.out‚0) disk./.usagePercent‚70.87301339177459% disk./.usageGB‚1056.0591163635254 disk.sda1.out‚0 disk.drbd1777.out‚0$ memory_percent‚96.62506580352783 proc.cpu.threads‚17 network.sent‚721549 network.recv‚235339 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 cpu‚25.104225719512776 disk.rbd1.out‚ 0.369140625$ disk.nvme0n1.out‚1511.3056640625! disk.dm-0.out‚1511.3056640625( proc.memory.availableMB‚ 552.94921875 disk.rbd2.out‚ 11.2333984375 disk.nvme0c0n1.in‚0.03125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 18.2109375 disk.drbd1777.in‚0 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.rbd2.in‚0 disk.rbd3.in‚0& disk.nvme0c0n1.out‚1511.3056640625 disk.sda3.in‚0 disk.sda.out‚ 18.2109375 disk.dm-0.in‚0.03125* proc.memory.percent‚16.984128952026367 disk.sda.in‚0´¦Z‘ø‚(:† àãÔîЫ™ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.926' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7852 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦Æf–ù‚(:ƒ àãÔÇ„ð› network.recv‚240866* proc.memory.percent‚16.984128952026367 network.sent‚771000 disk.rbd3.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 18.72265625$ memory_percent‚96.62423133850098 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 18.72265625 disk.sda.in‚0 disk.drbd1777.out‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd2.out‚ 11.2333984375 disk.sda1.out‚0 disk.dm-0.in‚0.03125$ disk./.usageGB‚1056.058925628662 disk.rbd2.in‚0 disk.sda.out‚ 18.72265625 disk.nvme0n1.in‚0.03125% disk.nvme0n1.out‚1610.65185546875' proc.memory.availableMB‚ 553.0859375" proc.memory.rssMB‚ 2782.6796875 cpu‚25.103059795728544 disk.nvme0c0n1.in‚0.03125" disk.dm-0.out‚1610.65185546875 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.369140625) disk./.usagePercent‚70.87300059139788 disk.rbd3.out‚0' disk.nvme0c0n1.out‚1610.65185546875 disk.sda3.in‚0 disk.drbd1777.in‚0ïTs­0ú+ _wandbruntime‚448  _runtime‚448ƒÕ®„Äû¾ _step‚48 & lr-AdamW/pg1‚5.423620268695385e-05 & lr-AdamW/pg2‚5.423620268695385e-05 trainer/global_step‚649 % _timestamp‚1.7884078383449047e+09 _runtime‚ 448.7571666738q¾ëÈü & lr-AdamW/pg1‚5.423620268695385e-05 & lr-AdamW/pg2‚5.423620268695385e-05 trainer/global_step‚649 % _timestamp‚1.7884078383449047e+09 _runtime‚ 448.757166673 _step‚480•bmh0ý+ _wandbruntime‚448  _runtime‚448M³¡²'þ¬O WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 904992801792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8237115 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚981 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚49 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19725636160512 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4803256 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202962944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8237117 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18642725084160 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚17737732282368 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 trainer/global_step‚649 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3575453 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18804629385216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21154784 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚17737750285312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8378707 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 921006775296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 905161034752 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 920803812352 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12917667 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚18642911320064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚17775509738990 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16872945482752 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 905161034752 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18804605445120 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16872928443392 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8237115 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3575992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _runtime‚ 448.757995735 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226903040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216¡F†“¶'ÿ°O K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8237115 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3575992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _runtime‚ 448.757995735 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226903040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 904992801792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8237115 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚981 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚49 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19725636160512 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4803256 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202962944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8237117 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18642725084160 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚17737732282368 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 trainer/global_step‚649 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3575453 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18804629385216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21154784 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚17737750285312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8378707 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 921006775296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 905161034752 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 920803812352 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12917667 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚18642911320064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚17775509738990 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16872945482752 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 905161034752 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18804605445120 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚168729284433921’>»a0€+ _wandbruntime‚449  _runtime‚449SB©b¶M E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚17802659932654 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8249774 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚18671387587072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚18671573836288 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1002 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8391759 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚17765019650048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚18671573836288 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21187256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 906385939968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18833518393344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 213435392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884078389971287e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12918914 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21168688 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3581660 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18833542333440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚17765019650048 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3581128 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16898888665088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 906554186240 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16898871625728 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚534 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4810631 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚981 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19755732697600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12918914 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚649 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 903953420990 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _runtime‚ 449.409683337 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4810633 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21187256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚17765001647104 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 922214304256 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8249776 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8249776 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19755970073088 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8249774 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 922427739648 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _step‚50 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21168688 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0<ˆÀ&‚ºM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚17802659932654 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8249774 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚18671387587072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚18671573836288 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1002 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8391759 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚17765019650048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚18671573836288 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21187256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 906385939968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18833518393344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 213435392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884078389971287e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12918914 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21168688 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3581660 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18833542333440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚17765019650048 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3581128 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16898888665088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 906554186240 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16898871625728 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚534 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4810631 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚981 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19755732697600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12918914 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚649 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 903953420990 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _runtime‚ 449.409683337 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4810633 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21187256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚17765001647104 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 922214304256 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8249776 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8249776 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19755970073088 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8249774 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 922427739648 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _step‚50 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21168688 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20075864643584 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 974664124416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?ùÛ£—àDeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13891829 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚21242156488192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8870065 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8870067 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 971870960790 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚18170104599552 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22780436 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8870067 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚19142140674198 % _timestamp‚1.7884078728188624e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 974664124416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚19101218522112 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5172106 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 211351552 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚19101200519168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3852711 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13910369 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13910369òGfܶ'‘°O PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13910369 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13910369 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20075864643584 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 974664124416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13891829 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚21242156488192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8870065 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8870067 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 971870960790 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚18170104599552 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22780436 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8870067 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚19142140674198 % _timestamp‚1.7884078728188624e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 974664124416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚19101218522112 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5172106 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 211351552 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚19101200519168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3852711 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4173332485À/p0’+ _wandbruntime‚483  _runtime‚483£3Í»&“µM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13930182 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13911616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8882724 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1002 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚19128469883904 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚19128487886848 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 976225508864 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9037188 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226889728 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22794340 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22794340 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3857707 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 973425239398 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13930182 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚19128469883904 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5179481 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 993292775936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚ 483.870126643 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9037716 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22812908 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚54 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20104713395712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21272482520576 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚18196030742528 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20104713395712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3858233 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚699 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19169473021286 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚981 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096ñ§Å\”¹M EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22794340 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3857707 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 973425239398 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13930182 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚19128469883904 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5179481 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 993292775936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚ 483.870126643 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9037716 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22812908 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚54 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20104713395712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21272482520576 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚18196030742528 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20104713395712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3858233 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚699 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19169473021286 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚981 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13930182 =~Ÿ©c7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13911616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8882724 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1002 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚19128469883904 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚19128487886848 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 976225508864 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9037188 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226889728 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22794340 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 49112540166º71µ0•+ _wandbruntime‚483  _runtime‚483{KŠ‹M–Ç .loss/FRt4/detection_loss‚4.726244313849343 0loss/SRqqt2/detection_loss‚4.084250853249901 _runtime‚ 483.871195677 _step‚55 .loss/FRt1/assignment_loss‚6.59760374262713 2loss/SRqqt2/assignment_loss‚3.3597256635364734 0loss/FBt2/assignment_loss‚0.6082935273696325 /loss/FRt3/detection_loss‚3.0121659672317596 1loss/SRqqt1/detection_loss‚1.9401559097526764 .loss/FBt1/detection_loss‚6.832073286175728 epoch‚0 trainer/global_step‚699 % _timestamp‚1.7884078734591477e+09 0loss/SRqqt4/detection_loss‚4.012450036779046 0loss/FBt4/assignment_loss‚0.7462883595573274 /loss/FRt2/assignment_loss‚7.274978848390801 /loss/FBt3/detection_loss‚1.8506613300083574 %loss/total_loss‚88.59039882236192 /loss/FRt3/assignment_loss‚7.113711604705224 0loss/SRqqt4/assignment_loss‚3.24982231259346 1loss/FBt3/assignment_loss‚0.46308269578329253 .loss/FRt1/detection_loss‚2.370892921651619 /loss/FRt2/detection_loss‚3.7152081450750662 0loss/SRqqt3/detection_loss‚1.685715032833347 /loss/FBt2/detection_loss‚2.8354933388026713 .loss/FBt4/detection_loss‚7.767951784165282 /loss/FRt4/assignment_loss‚7.308970928192139 1loss/SRqqt1/assignment_loss‚3.209084679340494 2loss/SRqqt3/assignment_loss‚3.2617091655731203 /loss/FBt1/assignment_loss‚0.563864375118018Ø–€Q—Ë 0loss/SRqqt2/detection_loss‚4.084250853249901 _runtime‚ 483.871195677 _step‚55 .loss/FRt1/assignment_loss‚6.59760374262713 2loss/SRqqt2/assignment_loss‚3.3597256635364734 0loss/FBt2/assignment_loss‚0.6082935273696325 /loss/FRt3/detection_loss‚3.0121659672317596 1loss/SRqqt1/detection_loss‚1.9401559097526764 .loss/FBt1/detection_loss‚6.832073286175728 epoch‚0 trainer/global_step‚699 % _timestamp‚1.7884078734591477e+09 0loss/SRqqt4/detection_loss‚4.012450036779046 0loss/FBt4/assignment_loss‚0.7462883595573274 /loss/FRt2/assignment_loss‚7.274978848390801 /loss/FBt3/detection_loss‚1.8506613300083574 %loss/total_loss‚88.59039882236192 /loss/FRt3/assignment_loss‚7.113711604705224 0loss/SRqqt4/assignment_loss‚3.24982231259346 1loss/FBt3/assignment_loss‚0.46308269578329253 .loss/FRt1/detection_loss‚2.370892921651619 /loss/FRt2/detection_loss‚3.7152081450750662 0loss/SRqqt3/detection_loss‚1.685715032833347 /loss/FBt2/detection_loss‚2.8354933388026713 .loss/FBt4/detection_loss‚7.767951784165282 /loss/FRt4/assignment_loss‚7.308970928192139 1loss/SRqqt1/assignment_loss‚3.209084679340494 2loss/SRqqt3/assignment_loss‚3.2617091655731203 /loss/FBt1/assignment_loss‚0.563864375118018 .loss/FRt4/detection_loss‚4.72624431384934379 ‘˜‚(:† ÌàãÔ¸ª¦™ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚120.418' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0836 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&Í™‚(:ý ÌàãÔ disk.rbd1.out‚ 0.369140625' disk.nvme0c0n1.out‚1810.23388671875 disk.nvme0n1.in‚0.03125" disk.dm-0.out‚1810.23388671875 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.sda1.out‚0 disk.drbd1777.out‚0) disk./.usagePercent‚70.87299495923213 disk.sda.out‚ 21.078125 disk.sda.in‚0% disk.nvme0n1.out‚1810.23388671875 disk.rbd2.out‚11.43310546875# memory_percent‚96.6299057006836 proc.cpu.threads‚17 network.sent‚839294 network.recv‚260275 disk.nvme0c0n1.in‚0.03125 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚0.03125 disk.dm-1.out‚ 21.078125% proc.memory.availableMB‚ 552.15625* proc.memory.percent‚16.984152793884277 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 21.078125 disk.rbd1.in‚0 cpu‚25.096390093879762 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 2782.68359375% disk./.usageGB‚1056.0588417053223Mù¿X›š‚(: ÛàãԀङ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚95.245' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.049000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸VMB›‚(:ý ÛàãÔ¹½Ôœ disk.rbd3.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚0.03125 disk.rbd0.in‚0 disk.rbd1.in‚0" disk.dm-0.out‚1909.51708984375) disk./.usagePercent‚70.87299495923213 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚21.84375* proc.memory.percent‚16.984200477600098 disk.sda2.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 0.369140625% disk.nvme0n1.out‚1909.51708984375 disk.dm-0.in‚0.03125 cpu‚25.095072635919596 disk.dm-1.in‚0# memory_percent‚96.6317892074585 proc.cpu.threads‚17 network.sent‚888769 network.recv‚265386% disk./.usageGB‚1056.0588417053223 disk.nvme0c0n1.in‚0.03125 disk.rbd3.out‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 551.84765625# proc.memory.rssMB‚ 2782.69140625 disk.rbd2.out‚13.08544921875' disk.nvme0c0n1.out‚1909.51708984375 disk.sda.out‚21.84375 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-1.out‚21.84375 disk.rbd2.in‚0šx0œ+ _wandbruntime‚517  _runtime‚517 ƒ_Ný _step‚56 & lr-AdamW/pg1‚6.259309062022872e-05 & lr-AdamW/pg2‚6.259309062022872e-05 trainer/global_step‚749 % _timestamp‚1.7884079071555674e+09 _runtime‚ 517.56771845é÷ÂAÇžÁ & lr-AdamW/pg2‚6.259309062022872e-05 trainer/global_step‚749 % _timestamp‚1.7884079071555674e+09 _runtime‚ 517.56771845 _step‚56 & lr-AdamW/pg1‚6.259309062022872e-0585ŠÃ0Ÿ+ _wandbruntime‚517  _runtime‚517"È" ¹' ³O G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1044335447040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9670443 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9669912 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚981 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1041507893054 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _runtime‚ 517.568720197 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1062720452096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9503017 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚749 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14884531 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚19467246676992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚21509190438912 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚20464668755968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4129487 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4128958 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚22758644659712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1044335447040 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚21509190438912 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚22758415659520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚20464686758912 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚21695924207616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚20464668755968 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚21509004203008 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚21695900267520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9503015 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14903071 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚20508589456190 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9503017 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5540954 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚21509004203008 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5540956 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚20464686758912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1002 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚19467263716352 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24387546 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9503015 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 205060096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1041342779198 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚57 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 205060096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1041342779198 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚57 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1044335447040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9670443 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9669912 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚981 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1041507893054 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _runtime‚ 517.568720197 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1062720452096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9503017 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚749 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14884531 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚19467246676992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚21509190438912 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚20464668755968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4129487 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4128958 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚22758644659712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1044335447040 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚21509190438912 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚22758415659520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚20464686758912 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚21695924207616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚20464668755968 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚21509004203008 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚21695900267520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9503015 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14903071 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚20508589456190 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9503017 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5540954 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚21509004203008 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5540956 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚20464686758912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1002 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚19467263716352 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24387546 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9503015 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1042731943742 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1045728585216 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚20491956123648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚ 518.226519519 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 trainer/global_step‚749 _step‚58 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9515676 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14904318 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9682840 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4135026 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14922884 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚20535921803070 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24419992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 [Ýä;0OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24419992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1045896831488 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9683359 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4134509 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚981 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚21724813215744 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 207143936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24438560 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚21537852955136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1064141570048 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚19493189859328 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚20536103956494 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1063934426112 $ _timestamp‚1.788407907814014e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5548333 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚19493206898688 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0úh*aÇ&¤ÁM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24419992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24419992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1045896831488 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9683359 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4134509 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚981 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚21724813215744 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 207143936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24438560 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚21537852955136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1064141570048 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚19493189859328 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚20536103956494 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1063934426112 $ _timestamp‚1.788407907814014e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5548333 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚19493206898688 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9515674 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14904318 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1045728585216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231084032 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚22788747641856 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1042731943742 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1045728585216 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚20491956123648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚ 518.226519519 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 trainer/global_step‚749 _step‚58 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9515676 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14904318 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9682840 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4135026 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14922884 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚20535921803070 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0:51\0¥+ _wandbruntime‚518  _runtime‚518µÿ¥O¦É 0loss/SRqqt2/detection_loss‚4.648390956900337 /loss/FBt1/detection_loss‚10.073024986187617 2loss/SRqqt2/assignment_loss‚3.0885530746344365 1loss/SRqqt3/detection_loss‚2.9991793398697726 /loss/FBt3/detection_loss‚1.8559390580898087 % _timestamp‚1.7884079078154771e+09 1loss/SRqqt3/assignment_loss‚3.253654246236764 2loss/SRqqt4/assignment_loss‚3.1417330529954697 /loss/FBt2/assignment_loss‚0.594095136335289 _step‚59 /loss/FRt4/assignment_loss‚6.833218221311216 1loss/SRqqt1/detection_loss‚2.1775184318489003 0loss/SRqqt4/detection_loss‚3.528082419104046 trainer/global_step‚749 /loss/FRt1/assignment_loss‚5.685360373872699 2loss/SRqqt1/assignment_loss‚2.4436527848243714 /loss/FRt3/detection_loss‚3.0431603023409846 -loss/FBt4/detection_loss‚9.04188349377364 /loss/FBt1/assignment_loss‚0.839447199611459 0loss/FBt4/assignment_loss‚0.7659506995696574 /loss/FRt1/detection_loss‚2.5568305868091006 /loss/FBt2/detection_loss‚1.8192226997399148 %loss/total_loss‚92.78549145132413 epoch‚0 /loss/FRt2/detection_loss‚6.6874597546458245 _runtime‚ 518.22747542 /loss/FRt2/assignment_loss‚7.055033149719239 /loss/FRt3/assignment_loss‚6.686996974945068 0loss/FBt3/assignment_loss‚0.2660911548226564 .loss/FRt4/detection_loss‚3.701013353135851iªŸìS§Í /loss/FRt3/assignment_loss‚6.686996974945068 0loss/FBt3/assignment_loss‚0.2660911548226564 .loss/FRt4/detection_loss‚3.701013353135851 0loss/SRqqt2/detection_loss‚4.648390956900337 /loss/FBt1/detection_loss‚10.073024986187617 2loss/SRqqt2/assignment_loss‚3.0885530746344365 1loss/SRqqt3/detection_loss‚2.9991793398697726 /loss/FBt3/detection_loss‚1.8559390580898087 % _timestamp‚1.7884079078154771e+09 1loss/SRqqt3/assignment_loss‚3.253654246236764 2loss/SRqqt4/assignment_loss‚3.1417330529954697 /loss/FBt2/assignment_loss‚0.594095136335289 _step‚59 /loss/FRt4/assignment_loss‚6.833218221311216 1loss/SRqqt1/detection_loss‚2.1775184318489003 0loss/SRqqt4/detection_loss‚3.528082419104046 trainer/global_step‚749 /loss/FRt1/assignment_loss‚5.685360373872699 2loss/SRqqt1/assignment_loss‚2.4436527848243714 /loss/FRt3/detection_loss‚3.0431603023409846 -loss/FBt4/detection_loss‚9.04188349377364 /loss/FBt1/assignment_loss‚0.839447199611459 0loss/FBt4/assignment_loss‚0.7659506995696574 /loss/FRt1/detection_loss‚2.5568305868091006 /loss/FBt2/detection_loss‚1.8192226997399148 %loss/total_loss‚92.78549145132413 epoch‚0 /loss/FRt2/detection_loss‚6.6874597546458245 _runtime‚ 518.22747542 /loss/FRt2/assignment_loss‚7.055033149719239;(­‚£¨‚(:… êàãÔäÅ› gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.285' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.057 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oS]´©‚(:ƒ êàãÔ—¼Õœ disk.sda.in‚0* proc.memory.percent‚16.984200477600098 disk.rbd1.out‚ 0.369140625# proc.memory.rssMB‚ 2782.69140625# disk./.usageGB‚1056.05717086792 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚13.08544921875 disk.nvme0c0n1.in‚0.03125 proc.cpu.threads‚17 disk.nvme0n1.in‚0.03125% disk.nvme0n1.out‚1911.67041015625 disk.dm-0.in‚0.03125 disk.rbd2.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 552.265625 disk.sda1.in‚0" disk.dm-0.out‚1911.67041015625 disk.sda3.out‚ 22.40234375 network.recv‚270718) disk./.usagePercent‚70.87288282793213 disk.sda.out‚ 22.40234375 disk.rbd0.out‚0 disk.rbd3.in‚0 cpu‚25.094295213296935 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 22.40234375' disk.nvme0c0n1.out‚1911.67041015625$ memory_percent‚96.62923812866211 network.sent‚895374pÑñ둪‚(:† ùàãÔê³£™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.983' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1966 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e/Mª«‚(:ú ùàãÔæɯœ disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.in‚0 network.sent‚945672 cpu‚25.092702947786496 disk.dm-1.in‚0) disk./.usagePercent‚70.87286081128418 disk.rbd1.out‚ 0.369140625 disk.rbd3.in‚0 disk.sda3.in‚0$ memory_percent‚96.63074016571045* proc.memory.percent‚16.984200477600098$ disk./.usageGB‚1056.056842803955 disk.nvme0n1.in‚0.03125! disk.nvme0n1.out‚ 2013.6015625 disk.dm-0.out‚ 2013.6015625 disk.dm-1.out‚ 24.60546875 disk.rbd0.out‚0# disk.nvme0c0n1.out‚ 2013.6015625 disk.sda.out‚ 24.60546875 disk.sda2.in‚0( proc.memory.availableMB‚ 552.01953125 proc.cpu.threads‚17 disk.rbd2.out‚13.22607421875 disk.rbd3.out‚0 disk.sda3.out‚ 24.60546875 disk.sda1.out‚0 disk.drbd1777.in‚0# proc.memory.rssMB‚ 2782.69140625 disk.dm-0.in‚0.03125 disk.nvme0c0n1.in‚0.03125 disk.drbd1777.out‚0 network.recv‚276854 disk.rbd1.in‚0ͱ0¬+ _wandbruntime‚552  _runtime‚552!éÄ­¾ & lr-AdamW/pg2‚6.677153458686615e-05 trainer/global_step‚799 % _timestamp‚1.7884079416071606e+09 _runtime‚ 552.019305813 _step‚60 & lr-AdamW/pg1‚6.677153458686615e-05Œ€Ò³È®Â % _timestamp‚1.7884079416071606e+09 _runtime‚ 552.019305813 _step‚60 & lr-AdamW/pg1‚6.677153458686615e-05 & lr-AdamW/pg2‚6.677153458686615e-05 trainer/global_step‚799DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15877233 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚21828136992768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1133316482048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 % _timestamp‚1.7884079416078885e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1133521542144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _step‚61 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1114175002624 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 229000192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚21828154995712 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22942329998336 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚20764405793792 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4406253 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10135967 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10316059 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23141571618816 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1110979711462 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10315484 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚21875220391398 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1114006769664 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26031740 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10135965 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚24275093160960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15895773 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22942143762432 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1114006769664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5909804 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15895773 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26013198 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23141547678720 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10135965 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 205060096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚21828154995712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0!2ÏÉî±¾O _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚21828154995712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22942329998336 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22942143762432 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚ 552.020266352 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚24274864160768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚575 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4405680 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15877233 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚21828136992768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1133316482048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 % _timestamp‚1.7884079416078885e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1133521542144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _step‚61 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1114175002624 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 229000192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚21828154995712 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22942329998336 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚20764405793792 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4406253 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10135967 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10316059 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23141571618816 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1110979711462 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10315484 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚21875220391398 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1114006769664 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26031740 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10135965 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚24275093160960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15895773 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22942143762432 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1114006769664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5909804 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15895773 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26013198 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23141547678720 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10135965 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 205060096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944=mrf0²+ _wandbruntime‚552  _runtime‚552íÉ&³ÃM ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚21855424360448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15897020 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 $ _timestamp‚1.788407942252357e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 552.665130924 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26064212 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15915586 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1112368876214 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚21855424360448 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚21902734891702 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚21855406357504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1134939092480 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5917183 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚62 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26064212 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10148624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202949632 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚21855406357504 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26045644 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5917181 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4411678 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22970806265344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10148626 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23170484567040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1115568154112 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10148626 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22970992514560 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10148624 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15915586 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1115568154112 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10328861 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226889728 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1115399907840 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚24305423659520 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1115399907840 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚24305196769792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚20790348976128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚23170460626944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1134736142848 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20790366015488 trainer/global_step‚799 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15897020 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10328333 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚21902552738278 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22970992514560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160¸Ûõ)Í&´ÇM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15897020 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10328333 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚21902552738278 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22970992514560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚21855424360448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15897020 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 $ _timestamp‚1.788407942252357e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 552.665130924 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26064212 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15915586 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1112368876214 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚21855424360448 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚21902734891702 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚21855406357504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1134939092480 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5917183 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚62 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26064212 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10148624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202949632 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚21855406357504 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26045644 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5917181 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4411678 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22970806265344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10148626 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23170484567040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1115568154112 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10148626 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22970992514560 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10148624 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15915586 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1115568154112 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10328861 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226889728 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1115399907840 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚24305423659520 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1115399907840 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚24305196769792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚20790348976128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚23170460626944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1134736142848 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20790366015488 trainer/global_step‚799>DxÊ0µ+ _wandbruntime‚552  _runtime‚552¯Z\&N¶È %loss/total_loss‚86.39463941307656 trainer/global_step‚799 % _timestamp‚1.7884079422540689e+09 /loss/FRt2/assignment_loss‚6.224312191917782 2loss/SRqqt1/assignment_loss‚2.6383647163709005 -loss/FBt4/detection_loss‚11.2688167989254 0loss/FBt4/assignment_loss‚0.5586191532560266 1loss/SRqqt2/detection_loss‚2.4039578951322116 0loss/SRqqt4/detection_loss‚4.910771045720939 /loss/FRt1/assignment_loss‚5.525542091440271 /loss/FRt4/assignment_loss‚6.814704612449363 -loss/FRt1/detection_loss‚3.20269548230701 2loss/SRqqt2/assignment_loss‚2.5419692635536193 2loss/SRqqt3/assignment_loss‚2.9864984273910524 1loss/FBt3/assignment_loss‚0.33611597227442463 .loss/FRt2/detection_loss‚3.948831226144518 /loss/FRt4/detection_loss‚3.2428519858254337 1loss/SRqqt1/detection_loss‚1.8988703026611569 _runtime‚ 552.666140456 2loss/SRqqt4/assignment_loss‚3.1242627158309473 _step‚63 1loss/FBt1/assignment_loss‚0.37950752634909196 .loss/FBt1/detection_loss‚6.209700547158718 epoch‚0 /loss/FRt3/assignment_loss‚5.705603412959887 1loss/SRqqt3/detection_loss‚2.1099273660747166 -loss/FBt2/detection_loss‚3.94188370845639 /loss/FBt2/assignment_loss‚0.587840795933612 .loss/FRt3/detection_loss‚3.485484482801479 .loss/FBt3/detection_loss‚2.347507692141617çieºR·Ì 2loss/SRqqt2/assignment_loss‚2.5419692635536193 2loss/SRqqt3/assignment_loss‚2.9864984273910524 1loss/FBt3/assignment_loss‚0.33611597227442463 .loss/FRt2/detection_loss‚3.948831226144518 /loss/FRt4/detection_loss‚3.2428519858254337 1loss/SRqqt1/detection_loss‚1.8988703026611569 _runtime‚ 552.666140456 2loss/SRqqt4/assignment_loss‚3.1242627158309473 _step‚63 1loss/FBt1/assignment_loss‚0.37950752634909196 .loss/FBt1/detection_loss‚6.209700547158718 epoch‚0 /loss/FRt3/assignment_loss‚5.705603412959887 1loss/SRqqt3/detection_loss‚2.1099273660747166 -loss/FBt2/detection_loss‚3.94188370845639 /loss/FBt2/assignment_loss‚0.587840795933612 .loss/FRt3/detection_loss‚3.485484482801479 .loss/FBt3/detection_loss‚2.347507692141617 %loss/total_loss‚86.39463941307656 trainer/global_step‚799 % _timestamp‚1.7884079422540689e+09 /loss/FRt2/assignment_loss‚6.224312191917782 2loss/SRqqt1/assignment_loss‚2.6383647163709005 -loss/FBt4/detection_loss‚11.2688167989254 0loss/FBt4/assignment_loss‚0.5586191532560266 1loss/SRqqt2/detection_loss‚2.4039578951322116 0loss/SRqqt4/detection_loss‚4.910771045720939 /loss/FRt1/assignment_loss‚5.525542091440271 /loss/FRt4/assignment_loss‚6.814704612449363 -loss/FRt1/detection_loss‚3.20269548230701? ¥A_œ¸‚(:‘ ˆáãÔƒº¹™ gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.906' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.781200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸÎ¹¹‚(:ù ˆáãÔõ¤ý› network.recv‚280398 disk.nvme0n1.in‚0.03125! disk.nvme0n1.out‚ 2112.6640625 disk.dm-0.in‚0.03125# disk.nvme0c0n1.out‚ 2112.6640625 disk lK.sda3.out‚ 25.00390625' proc.memory.availableMB‚ 552.0234375 disk.dm-0.out‚ 2112.6640625 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 0.369140625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚0.03125 network.sent‚951259 disk.rbd2.out‚13.22607421875 disk.sda.out‚ 25.00390625 disk.sda1.in‚0* proc.memory.percent‚16.984200477600098 disk.rbd1.in‚0 disk.dm-1.out‚ 25.00390625 disk.sda2.out‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2782.69140625 cpu‚25.09163358118549) disk./.usagePercent‚70.87287847580406% disk./.usageGB‚1056.0571060180664 disk.dm-1.in‚0 disk.drbd1777.in‚0$ memory_percent‚96.63071632385254 proc.cpu.threads‚17%OmÒ‘º‚(:† —áãÔ¹˜œ™ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.983' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9966 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“ÿ,R»‚(:„ —áãÔ骛 disk.rbd3.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚0.03125 disk.rbd0.out‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2782.69140625" disk.dm-0.out‚2113.50146484375 disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd1777.in‚0# memory_percent‚96.6299057006836) disk./.usagePercent‚70.87287719576638 disk.nvme0c0n1.in‚0.03125* proc.memory.percent‚16.984200477600098# disk./.usageGB‚1056.05708694458 disk.rbd2.out‚ 13.33203125 disk.sda2.in‚0 disk.sda3.out‚ 25.65234375 disk.drbd1777.out‚0% proc.memory.availableMB‚ 552.15625 network.sent‚ 1.000146e+06 disk.rbd2.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚0.03125' disk.nvme0c0n1.out‚2113.50146484375 disk.sda1.out‚0 disk.rbd1.out‚ 0.369140625% disk.nvme0n1.out‚2113.50146484375 cpu‚25.090640270498163 network.recv‚285350 disk.dm-1.in‚0 disk.dm-1.out‚ 25.65234375 disk.sda.out‚ 25.65234375 disk.rbd0.in‚0Îö,œ¼‚(:‘ ¦áãÔÙ§«™ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.821' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.164199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J/Ž¦½‚(:ù ¦áãÔãûÀ› disk.rbd0.out‚0 network.recv‚290596% disk./.usageGB‚1056.0571327209473 disk.dm-0.in‚0.03125" disk.dm-0.out‚2212.99365234375 disk.rbd2.out‚14.90625 disk.nvme0c0n1.in‚0.03125 disk.sda.in‚0% disk.nvme0n1.out‚2212.99365234375 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2782.69140625 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd3.in‚0' disk.nvme0c0n1.out‚2212.99365234375 disk.sda2.in‚0 disk.sda.out‚26.1875' proc.memory.availableMB‚ 552.3984375* proc.memory.percent‚16.984200477600098 disk.rbd1.in‚0 disk.sda3.in‚0 disk.sda3.out‚26.1875 disk.nvme0n1.in‚0.03125 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda1.in‚0$ memory_percent‚96.62842750549316( disk./.usagePercent‚70.8728802678568 disk.dm-1.out‚26.1875 disk.sda2.out‚0 cpu‚25.089625399713633 network.sent‚ 1.006806e+06 disk.rbd1.out‚ 0.369140625 disk.drbd1777.in‚0 ¿#a0¾+ _wandbruntime‚586  _runtime‚586ÏIc1Ä¿¾ _runtime‚ 586.436741697 _step‚64 & lr-AdamW/pg1‚7.094997855350358e-05 & lr-AdamW/pg2‚7.094997855350358e-05 trainer/global_step‚849 % _timestamp‚1.7884079760246973e+09ûà¥ðÈÀ _step‚64 & lr-AdamW/pg1‚7.094997855350358e-05 & lr-AdamW/pg2‚7.094997855350358e-05 trainer/global_step‚849 % _timestamp‚1.7884079760246973e+09 _runtime‚ 586.436741697@²ï¾0Á+ _wandbruntime‚586  _runtime‚586‘AãoÀ'ºO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16869935 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27638850 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10768915 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1204195992576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚24375469557760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884079760253851e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16869935 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6278656 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1183846325248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10768915 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚23191623232512 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233194496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚24587195089920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚24587219030016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1183678092288 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚23191605229568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1180286416014 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚23241851326606 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4681857 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚23191605229568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16888475 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1204405246976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6278654 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209254400 trainer/global_step‚849 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24375283321856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1183846325248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24375469557760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚25791391082496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16888475 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4681332 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚23242033479822 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1180451529870ýV[°Ä'þO Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24375283321856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1183846325248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24375469557760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚25791391082496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16888475 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4681332 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚23242033479822 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1180451529870 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16869935 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27638850 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10768915 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1204195992576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚24375469557760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884079760253851e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16869935 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6278656 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1183846325248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10768915 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚23191623232512 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233194496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚24587195089920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚24587219030016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1183678092288 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚23191605229568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1180286416014 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚23241851326606 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4681857 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚23191605229568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16888475 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1204405246976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6278654 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209254400 trainer/global_step‚849AÌæ 0Ä+ _wandbruntime‚587  _runtime‚587‹!œrzÅÄM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1185239476736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27671296 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22087508092928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16908288 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚23218874594304 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚23269365826910 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10781576 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22087525132288 trainer/global_step‚849 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚24616108038144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚23269183673486 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27689864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1181675580558 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚23218874594304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1185071230464 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1185071230464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27689864 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6286031 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1181840694622 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚23218892597248 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10972868 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25821721541120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10781574 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16889722 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6286033 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1205822744064 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1205613502976 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 % _timestamp‚1.7884079766799903e+09 QCDeviceStatsMonitor.on_train_batch_end²” íP/inactive_split_bytes.all.peak‚ 262255616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚66 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16908288 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚25821954722304 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚513 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10972868 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25821721541120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10781574 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16889722 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6286033 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1205822744064 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1205613502976 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 % _timestamp‚1.7884079766799903e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚66 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16908288 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚25821954722304 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚513 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16908288 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚23218874594304 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚23269365826910 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10781576 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22087525132288 trainer/global_step‚849 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚24616108038144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚23269183673486 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27689864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1181675580558 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚23218874594304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1185071230464 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1185071230464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27689864 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6286031 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1181840694622 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚23218892597248 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24555091469312 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200865792 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚27307896497664 _step‚69 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6647506 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚24555073466368 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11602668 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27308121303552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29264502 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚25808422881280 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1253349414912 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1275053996544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚23358741066752 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17881177 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 # _timestamp‚1.78840801017988e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚24608482261814 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6647504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26032842501120 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚24608664415030 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚24555091469312 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚25808609117184 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17862637 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1249923348278 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26032866441216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1253517647872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29283044 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224805888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11401867 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11401867 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚25808609117184 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚24555073466368 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11602146 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1275254862336 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1253517647872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248EB‹F­Ó»O TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11401867 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11401867 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚25808609117184 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚24555073466368 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11602146 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1275254862336 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1253517647872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17862637 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11401865 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4954642 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚25808422881280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24555091469312 ]ODeviceStatsMonitor.on_train_batch_start/iTv|Ènactive_split_bytes.small_pool.current‚ 200865792 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚27307896497664 _step‚69 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6647506 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚24555073466368 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11602668 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27308121303552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29264502 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚25808422881280 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1253349414912 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1275053996544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚23358741066752 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17881177 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 # _timestamp‚1.78840801017988e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚24608482261814 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6647504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26032842501120 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚24608664415030 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚24555091469312 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚25808609117184 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17862637 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1249923348278 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26032866441216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1253517647872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29283044 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224805888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360EñbÞˆ0Ô+ _wandbruntime‚621  _runtime‚621Vò-ÜÊ&ÕÄM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11414526 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11414526 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29296948 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11615680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17882424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1276679841280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11414524 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17900990 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚24582360834048 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4960267 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚530 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11615148 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6654881 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27338459230720 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚24635814608694 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17882424 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚25837271633408 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1254910799360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4960797 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚23384667209728 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1251147398966 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29315516 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚24582342831104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26061779389440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228986880 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚23384684249088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 trainer/global_step‚899 % _timestamp‚1.7884080108378983e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1254910799360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199ØÇ0ZÎ&ÖÈM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚24582342831104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26061779389440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228986880 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚23384684249088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 trainer/global_step‚899 % _timestamp‚1.7884080108378983e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1254910799360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11414526 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11414526 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29296948 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11615680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17882424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1276679841280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11414524 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17900990 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚24582360834048 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4960267 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚530 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11615148 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6654881 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27338459230720 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚24635814608694 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17882424 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚25837271633408 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1254910799360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4960797 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚23384667209728 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1251147398966 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29315516 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416F^U30×+ _wandbruntime‚621  _runtime‚621Þq:wOØÉ _runtime‚ 621.251697705 _step‚71 trainer/global_step‚899 %loss/total_loss‚89.39391281725122 1loss/FBt3/assignment_loss‚0.30814075147571535 .loss/FRt3/detection_loss‚4.013121347874403 /loss/FBt2/detection_loss‚12.728754800559248 0loss/FBt4/assignment_loss‚0.5434569304509621 /loss/FRt2/assignment_loss‚6.152320872182432 1loss/SRqqt1/assignment_loss‚2.412387749552727 1loss/SRqqt4/assignment_loss‚3.287115788459778 1loss/SRqqt1/detection_loss‚2.2849767292821843 0loss/SRqqt2/detection_loss‚2.899796718588242 0loss/SRqqt3/detection_loss‚1.349144213408231 /loss/FBt4/detection_loss‚10.178139082023076 % _timestamp‚1.7884080108396032e+09 epoch‚0 0loss/FBt1/assignment_loss‚0.2131591122573985 .loss/FRt4/assignment_loss‚6.61851323884109 2loss/SRqqt3/assignment_loss‚3.0015179953154396 0loss/SRqqt4/detection_loss‚3.264543350338936 .loss/FBt3/detection_loss‚1.600555644801304 /loss/FRt2/detection_loss‚3.5886170844668928 /loss/FRt4/detection_loss‚3.8071299709122757 .loss/FBt1/detection_loss‚4.761325742929213 0loss/FBt2/assignment_loss‚0.5668707758188247 /loss/FRt1/detection_loss‚2.7954128409018284 /loss/FRt1/assignment_loss‚5.684379608904729 /loss/FRt3/assignment_loss‚4.752271848917007 1loss/SRqqt2/assignment_loss‚2.582260618989284 íSSÙÍ 0loss/FBt1/assignment_loss‚0.2131591122573985 .loss/FRt4/assignment_loss‚6.61851323884109 2loss/SRqqt3/assignment_loss‚3.0015179953154396 0loss/SRqqt4/detection_loss‚3.264543350338936 .loss/FBt3/detection_loss‚1.600555644801304 /loss/FRt2/detection_loss‚3.5886170844668928 /loss/FRt4/detection_loss‚3.8071299709122757 .loss/FBt1/detection_loss‚4.761325742929213 0loss/FBt2/assignment_loss‚0.5668707758188247 /loss/FRt1/detection_loss‚2.7954128409018284 /loss/FRt1/assignment_loss‚5.684379608904729 /loss/FRt3/assignment_loss‚4.752271848917007 1loss/SRqqt2/assignment_loss‚2.582260618989284 _runtime‚ 621.251697705 _step‚71 trainer/global_step‚899 %loss/total_loss‚89.39391281725122 1loss/FBt3/assignment_loss‚0.30814075147571535 .loss/FRt3/detection_loss‚4.013121347874403 /loss/FBt2/detection_loss‚12.728754800559248 0loss/FBt4/assignment_loss‚0.5434569304509621 /loss/FRt2/assignment_loss‚6.152320872182432 1loss/SRqqt1/assignment_loss‚2.412387749552727 1loss/SRqqt4/assignment_loss‚3.287115788459778 1loss/SRqqt1/detection_loss‚2.2849767292821843 0loss/SRqqt2/detection_loss‚2.899796718588242 0loss/SRqqt3/detection_loss‚1.349144213408231 /loss/FBt4/detection_loss‚10.178139082023076 % _timestamp‚1.7884080108396032e+09 epoch‚0G\­ÚœÚ‚(:‘ ÓáãÔøÉ™ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.158' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.431600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nŽ‹ Û‚(: ÓáãÔÚÔœ disk.drbd1777.in‚0 disk.rbd1.out‚ 0.369140625 disk.rbd2.out‚15.13720703125 proc.cpu.threads‚17" disk.dm-0.out‚2413.67333984375$ memory_percent‚96.62802219390869( proc.memory.availableMB‚ 552.46484375" proc.memory.rssMB‚ 2782.6953125 disk.rbd0.in‚0 disk.sda1.in‚0) disk./.usagePercent‚70.87287233162323 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚27.96875 disk.drbd1777.out‚0 disk.nvme0n1.in‚0.03125 network.recv‚304556 disk.sda3.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚0.03125 disk.rbd3.out‚0* proc.memory.percent‚16.984224319458008 disk.sda1.out‚0 disk.sda.in‚0 cpu‚25.08681025629799 disk.sda.out‚27.96875% disk.nvme0n1.out‚2413.67333984375 disk.rbd2.in‚0 disk.dm-1.out‚27.96875 network.sent‚ 1.068579e+06 disk.rbd3.in‚0 disk.nvme0c0n1.in‚0.03125 disk.dm-1.in‚0 disk.rbd0.out‚0$ disk./.usageGB‚1056.057014465332' disk.nvme0c0n1.out‚2413.67333984375'¬}óÜ‚(:… âáãÔƒÅß™ gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.415' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.483 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#0NÝ‚(:‰ âáãÔý·Šœ disk.rbd2.out‚15.15283203125 disk.dm-1.out‚ 28.19140625' disk.nvme0c0n1.out‚2512.75927734375 disk.sda1.out‚0( proc.memory.availableMB‚ 552.92578125 disk.rbd0.out‚0 disk.sda.in‚0" disk.dm-0.out‚2512.75927734375) disk./.usagePercent‚70.87287130759309 disk.drbd1777.in‚0 disk.rbd1.out‚ 0.369140625 disk.nvme0n1.in‚0.03125 network.recv‚310454$ disk./.usageGB‚1056.056999206543 disk.rbd3.in‚0 disk.sda.out‚ 28.19140625 disk.drbd1777.out‚0 disk.rbd1.in‚0 cpu‚25.08602983973014 proc.cpu.threads‚17 network.sent‚ 1.118541e+06 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚0.03125# memory_percent‚96.6252088546753 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 28.19140625 disk.sda1.in‚0% disk.nvme0n1.out‚2512.75927734375" proc.memory.rssMB‚ 2782.6953125 disk.dm-0.in‚0.03125* proc.memory.percent‚16.984224319458008>Úç0Þ+ _wandbruntime‚655  _runtime‚655°º-VÃß½ trainer/global_step‚949 % _timestamp‚1.7884080448701444e+09 _runtime‚ 655.28227789 _step‚72 & lr-AdamW/pg1‚7.930686648677844e-05 & lr-AdamW/pg2‚7.930686648677844e-05t¦GÇàÁ & lr-AdamW/pg2‚7.930686648677844e-05 trainer/global_step‚949 % _timestamp‚1.7884080448701444e+09 _runtime‚ 655.28227789 _step‚72 & lr-AdamW/pg1‚7.930686648677844e-05HÑ@\¼0á+ _wandbruntime‚655  _runtime‚655 æ1Ô¬âºO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚27241562440704 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚25918559706112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚27241748676608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1323188970496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1319230052830 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27241562440704 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚25918541703168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚523 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7016356 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 trainer/global_step‚949 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚27241748676608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30908696 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30890154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚25975295350238 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚24655900183552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233194496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1345889649152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982"º}Ä'ã¾O M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30908696 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30890154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚25975295350238 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚24655900183552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233194496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1345889649152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚27241562440704 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚25918559706112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚27241748676608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1323188970496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1319230052830 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27241562440704 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚25918541703168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚523 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7016356 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 trainer/global_step‚949 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚27241748676608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2Ia`°«0ä+ _wandbruntime‚655  _runtime‚655K…uÉ&åÃM $ _timestamp‚1.788408045516149e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30941168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚74 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚25945811067904 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1320784331438 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30922600 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚25945811067904 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30941168 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28854708789248 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚26002445543902 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18875126 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12261526 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚25945829070848 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18893692 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1347305928704 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚26002627697326 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12047474 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233181184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12047476 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12047476 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1320619217374 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7023731 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5238321 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18893692 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199gÐ æÇM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12047474 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 $ _timestamp‚1.788408045516149e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30941168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚74 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚25945811067904 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1320784331438 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30922600 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚25945811067904 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30941168 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28854708789248 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚26002445543902 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18875126 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12261526 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚25945829070848 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18893692 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1347305928704 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚26002627697326 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12047474 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233181184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12047476 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12047476 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1320619217374 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7023731 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5238321 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18893692 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 EC í‚(:ÿ €âãÔ»êÁœ network.recv‚320025$ disk./.usageGB‚1056.057415008545 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚16.80126953125 disk.sda2.in‚0" proc.memory.rssMB‚ 2782.6953125 disk.rbd2.in‚0 disk.sda3.out‚ 29.41796875& proc.memory.availableMB‚ 552.578125 cpu‚25.084096358927884 proc.cpu.threads‚17) disk./.usagePercent‚70.87289921241432 disk.rbd1.in‚0 disk.dm-0.out‚ 2613.01953125 disk.dm-1.out‚ 29.41796875 disk.sda2.out‚0 network.sent‚ 1.174033e+06 disk.sda.out‚ 29.41796875 disk.sda1.out‚0 disk.rbd1.out‚ 0.517578125 disk.dm-0.in‚0.03125 disk.rbd3.out‚0 disk.sda3.in‚0 disk.sda.in‚0$ disk.nvme0c0n1.out‚ 2613.01953125* proc.memory.percent‚16.984224319458008" disk.nvme0n1.out‚ 2613.01953125 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.nvme0c0n1.in‚0.03125# memory_percent‚96.6273307800293 disk.drbd1777.out‚0 disk.nvme0n1.in‚0.03125 disk.rbd0.out‚0­èn0î+ _wandbruntime‚689  _runtime‚689àG©.Ãï½ $ _timestamp‚1.788408079226991e+09 _runtime‚ 689.639161261 _step‚76 & lr-AdamW/pg1‚8.348531045341588e-05 & lr-AdamW/pg2‚8.348531045341588e-05 trainer/global_step‚999p€ÎÇðÁ & lr-AdamW/pg1‚8.348531045341588e-05 & lr-AdamW/pg2‚8.348531045341588e-05 trainer/global_step‚999 $ _timestamp‚1.788408079226991e+09 _runtime‚ 689.639161261 _step‚76LhÍùw0ñ+ _wandbruntime‚689  _runtime‚689G‡¬àÀ'òºO dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚28924161263616 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1416958222848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12891400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28924137323520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28674888236032 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5506746 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7385204 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27282027942912 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32534348 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1392860293120 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚77 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚999 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1388701871238 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32515806 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12667765 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27282009939968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚30340890486272 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7385206 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1392860293120 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚25953042260992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1392692060160 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27341926285446 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12667767 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12891952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12667767 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7884080792277534e+09 _runtime‚ 689.640134755 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1392692060160 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28674702000128 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19866581 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1416753162752 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19848041 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28674702000128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚25953059300352 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚550 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 205060096 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27282009939968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚25953042260992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1392692060160 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27341926285446 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12667767 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12891952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12667767 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7884080792277534e+09 _runtime‚ 689.640134755 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1392692060160 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28674702000128 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19866581 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1416753162752 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19848041 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28674702000128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚25953059300352 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚550 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 205060096 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27282009939968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28674888236032 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5506746 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7385204 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27282027942912 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32534348 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1392860293120 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚77 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚999 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1388701871238 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32515806 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12667765 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_poæÞªm´ol.freed‚27282009939968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚30340890486272 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7385206Mù»›ô‚(: âãÔìס™ gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚95.356' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.071199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XRûõ‚(:ð âãÔÒùÄœ disk.dm-1.in‚0( proc.memory.availableMB‚ 552.58984375 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda2.in‚0 disk.drbd1777.out‚0" proc.memory.rssMB‚ 2782.6953125) disk./.usagePercent‚70.87289921241432$ disk./.usageGB‚1056.057415008545 disk.rbd0.in‚0 disk.rbd2.out‚16.86376953125 disk.sda2.out‚0 disk.sda.in‚0 network.sent‚ 1.179819e+06 disk.rbd3.in‚0 disk.nvme0c0n1.in‚0.03125" disk.dm-0.out‚2712.20166015625$ memory_percent‚96.62725925445557* proc.memory.percent‚16.984224319458008 cpu‚25.083728672141174' disk.nvme0c0n1.out‚2712.20166015625 disk.sda3.in‚0 disk.sda.out‚30 disk.rbd1.out‚ 0.517578125 disk.dm-0.in‚0.03125 network.recv‚323867 disk.rbd0.out‚0% disk.nvme0n1.out‚2712.20166015625 disk.dm-1.out‚30 disk.sda3.out‚30 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚0.03125 disk.sda1.in‚0 disk.drbd1777.in‚0lÔi0ö+ _wandbruntime‚690  _runtime‚690ÚmsÊ&÷ÄM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32548252 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚25979002482688 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1418165397504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19867828 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28703550752256 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5512209 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32566820 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1394253444608 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚27309297307648 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32548252 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12904242 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 239472640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚78 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚550 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12904792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28703550752256 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1394085198336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚27309279304704 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7392583 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1418380930048 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7392581 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1390256149846 % _timestamp‚1.7884080798736384e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 222112256 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1390091035782 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27369076479110 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12680424 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27309297307648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚548 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28953074211840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32566820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1394253444608 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5511661 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚ 690.286394156Œ%5Î&øÈM H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚27309279304704 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7392583 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1418380930048 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7392581 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1390256149846 % _timestamp‚1.7884080798736384e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 222112256 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1390091035782 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27369076479110 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12680424 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27309297307648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚548 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28953074211840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32566820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1394253444608 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5511661 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚ 690.286394156 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32548252 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚25979002482688 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1418165397504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19867828 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28703550752256 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5512209 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32566820 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1394253444608 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚27309297307648 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32548252 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12904242 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 239472640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚78 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚550 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12904792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28703550752256 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1394085198336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416N¼µ0Q0ù+ _wandbruntime‚690  _runtime‚690¶ù–ÙRúÌ /loss/FRt2/assignment_loss‚5.500255609045223 1loss/SRqqt4/assignment_loss‚3.211877839905875 .loss/FBt1/detection_loss‚6.546578996560791 trainer/global_step‚999 /loss/FRt3/assignment_loss‚4.524215990642332 0loss/FRt4/assignment_loss‚6.5566906332969666 1loss/SRqqt1/assignment_loss‚2.439539150280111 1loss/SRqqt2/assignment_loss‚2.371924682608191 1loss/FBt1/assignment_loss‚0.30633030673859385 1loss/FBt3/assignment_loss‚0.17391383365844376 .loss/FRt3/detection_loss‚2.939990018336278 epoch‚0 1loss/SRqqt3/assignment_loss‚2.811459256708622 0loss/FBt4/assignment_loss‚0.8387959414006522 /loss/FRt2/detection_loss‚3.2393513790198734 .loss/FRt4/detection_loss‚3.383183352649212 1loss/SRqqt2/detection_loss‚2.9557951812474235 1loss/SRqqt4/detection_loss‚3.6615430549496697 1loss/SRqqt3/detection_loss‚1.1570007153656925 _runtime‚ 690.287615577 _step‚79 /loss/FBt2/detection_loss‚11.457458351339612 /loss/FBt3/detection_loss‚1.4134462072821836 %loss/total_loss‚88.94781447668967 0loss/FRt1/assignment_loss‚5.5228170024024115 .loss/FRt1/detection_loss‚3.028996256214601 0loss/SRqqt1/detection_loss‚2.549489109621017 .loss/FBt4/detection_loss‚11.28990268955628 0loss/FBt2/assignment_loss‚1.0672589178596223 % _timestamp‚1.7884080798755481e+09˜&ÙçVûÐ 1loss/SRqqt3/detection_loss‚1.1570007153656925 _runtime‚ 690.287615577 _step‚79 /loss/FBt2/detection_loss‚11.457458351339612 /loss/FBt3/detection_loss‚1.4134462072821836 %loss/total_loss‚88.94781447668967 0loss/FRt1/assignment_loss‚5.5228170024024115 .loss/FRt1/detection_loss‚3.028996256214601 0loss/SRqqt1/detection_loss‚2.549489109621017 .loss/FBt4/detection_loss‚11.28990268955628 0loss/FBt2/assignment_loss‚1.0672589178596223 % _timestamp‚1.7884080798755481e+09 /loss/FRt2/assignment_loss‚5.500255609045223 1loss/SRqqt4/assignment_loss‚3.211877839905875 .loss/FBt1/detection_loss‚6.546578996560791 trainer/global_step‚999 /loss/FRt3/assignment_loss‚4.524215990642332 0loss/FRt4/assignment_loss‚6.5566906332969666 1loss/SRqqt1/assignment_loss‚2.439539150280111 1loss/SRqqt2/assignment_loss‚2.371924682608191 1loss/FBt1/assignment_loss‚0.30633030673859385 1loss/FBt3/assignment_loss‚0.17391383365844376 .loss/FRt3/detection_loss‚2.939990018336278 epoch‚0 1loss/SRqqt3/assignment_loss‚2.811459256708622 0loss/FBt4/assignment_loss‚0.8387959414006522 /loss/FRt2/detection_loss‚3.2393513790198734 .loss/FRt4/detection_loss‚3.383183352649212 1loss/SRqqt2/detection_loss‚2.9557951812474235 1loss/SRqqt4/detection_loss‚3.6615430549496697O—òK´‘ü‚(:† žâãÔŒž³™ gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.214' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2428 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë ý‚(:Œ žâãÔ÷úÂœ network.sent‚ 1.218183e+06 disk.rbd0.in‚0 disk.rbd2.out‚16.87939453125 disk.sda1.in‚0 disk.drbd1777.in‚0# memory_percent‚96.6289758682251" proc.memory.rssMB‚ 2782.6953125 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.drbd1777.out‚0) disk./.usagePercent‚70.87290356454241' disk.nvme0c0n1.out‚2712.54931640625 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 30.56640625 disk.rbd1.out‚ 0.5498046875% disk.nvme0n1.out‚2712.54931640625( proc.memory.availableMB‚ 552.30859375 proc.cpu.threads‚17 network.recv‚329565 disk.rbd3.out‚0 disk.sda1.out‚0 disk.dm-0.in‚0.03125 disk.sda2.in‚0 disk.nvme0n1.in‚0.03125% disk./.usageGB‚1056.0574798583984 disk.nvme0c0n1.in‚0.03125 disk.sda.out‚ 30.56640625" disk.dm-0.out‚2712.54931640625* proc.memory.percent‚16.984224319458008 cpu‚25.082693639395877 disk.dm-1.out‚ 30.56640625 disk.rbd0.out‚0Ix¢þ‚(:„ ­âãÔæì¾™ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.87' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.974 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01܃Îÿ‚(:‡ ­âãԹ逜 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚0.03125! disk.dm-0.out‚2811.9736328125 disk.rbd0.out‚0$ disk./.usageGB‚1056.057460784912 disk.sda3.out‚ 30.98828125 disk.rbd3.out‚0 disk.rbd2.out‚ 17.0322265625 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 network.recv‚334306* proc.memory.percent‚16.984224319458008 cpu‚25.08173469366687) disk./.usagePercent‚70.87290228450473 disk.sda2.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 0.5498046875 disk.dm-1.out‚ 30.98828125( proc.memory.availableMB‚ 552.02734375 network.sent‚ 1.257341e+06 disk.sda.in‚0 disk.dm-0.in‚0.03125 disk.dm-1.in‚0$ memory_percent‚96.63069248199463 disk.nvme0c0n1.in‚0.03125 disk.sda.out‚ 30.98828125$ disk.nvme0n1.out‚2811.9736328125 disk.rbd0.in‚0 disk.rbd3.in‚0" proc.memory.rssMB‚ 2782.6953125& disk.nvme0c0n1.out‚2811.9736328125 disk.drbd1777.in‚0 proc.cpu.threads‚17jOÉü0€+ _wandbruntime‚724  _runtime‚724j—-íý % lr-AdamW/pg1‚8.76637544200533e-05 % lr-AdamW/pg2‚8.76637544200533e-05 trainer/global_step‚1049 % _timestamp‚1.7884081136900046e+09 _runtime‚ 724.102269064 _step‚80%¼¹òÇ‚Á % _timestamp‚1.7884081136900046e+09 _runtime‚ 724.102269064 _step‚80 % lr-AdamW/pg1‚8.76637544200533e-05 % lr-AdamW/pg2‚8.76637544200533e-05 trainer/global_step‚1049PÅRBG0ƒ+ _wandbruntime‚724  _runtime‚724ÀÍÛJe„»O \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚28645496179712 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚28645478176768 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30108027795456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13300715 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1462531615744 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5780942 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚28708375067438 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚490 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7754054 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚31857441305600 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13300717 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34160000 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30107841559552 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218514432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1458173689646 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.fre ·GŠ\ed‚30107841559552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884081136908185e+09 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚27250218417152 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1462363382784 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13300715 trainer/global_step‚1049 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20859283 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20840743 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 724.103295207 _step‚81 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30369808674816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31857659820032 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1458338803502 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194574336 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30108027795456 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20859283 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1462531615744 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1462363382784 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13300717 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13535486 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1487851145216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20840743 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚28645478176768 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚28708557220654 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460€—ïÕÅ'…¿O A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30107841559552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884081136908185e+09 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚27250218417152 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1462363382784 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13300715 trainer/global_step‚1049 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20859283 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20840743 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 724.103295207 _step‚81 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30369808674816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31857659820032 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1458338803502 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 194574336 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30108027795456 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20859283 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1462531615744 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1462363382784 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13300717 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13535486 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1487851145216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20840743 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚28645478176768 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚28708557220654 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚28645496179712 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚28645478176768 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30108027795456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13300715 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1462531615744 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5780942 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚28708375067438 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚490 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7754054 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚31857441305600 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13300717 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34160000 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30107841559552 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218514432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1458173689646 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20860530 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30136504062464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1489074441728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20879096 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5786787 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚30136690311680 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28672747541504 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1463756520960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13548220 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _step‚82 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34192472 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34192472 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13313376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1463924767232 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31887772124672 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13313374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30398721623040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 trainer/global_step‚1049 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1463756520960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 205046784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚ 724.755627607 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1459727968254 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13547688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚530 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 222112256 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34173904 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27276144560128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1459562854190Ô.ÁˆÉM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1459727968254 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13547688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚530 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 222112256 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34173904 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27276144560128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1459562854190 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20860530 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30136504062464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1489074441728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20879096 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5786787 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚30136690311680 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28672747541504 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1463756520960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13548220 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _step‚82 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34192472 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34192472 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13313376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1463924767232 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31887772124672 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13313374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30398721623040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 trainer/global_step‚1049 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1463756520960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 205046784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚ 724.755627607 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709R¥šh0‰+ _wandbruntime‚724  _runtime‚724ÀzSŠÍ /loss/FBt1/detection_loss‚4.2455385133172525 epoch‚0 .loss/FRt1/detection_loss‚4.774443692144225 /loss/FRt2/detection_loss‚3.0816158509025207 %loss/total_loss‚82.87383816651194 .loss/FRt2/assignment_loss‚5.48907883809163 1loss/SRqqt2/assignment_loss‚2.857789562103596 0loss/FBt4/assignment_loss‚0.4950204354900052 % _timestamp‚1.7884081143448708e+09 _runtime‚ 724.756926024 1loss/SRqqt3/detection_loss‚0.9581430883859707 trainer/global_step‚1049 /loss/FRt3/assignment_loss‚4.625845318255217 2loss/SRqqt3/assignment_loss‚2.7158921232706383 1loss/SRqqt4/assignment_loss‚3.294554454524343 0loss/FBt1/assignment_loss‚0.3111364713430786 0loss/FBt2/assignment_loss‚0.5042653986149364 1loss/FBt3/assignment_loss‚0.20901517895364316 /loss/FRt4/assignment_loss‚6.324271869659424 2loss/SRqqt1/assignment_loss‚2.8929188777418697 .loss/FRt4/detection_loss‚2.937986143430074 .loss/FBt2/detection_loss‚7.287977516651154 /loss/FRt3/detection_loss‚3.3113643222528957 1loss/SRqqt1/detection_loss‚3.4892006432361464 _step‚83 /loss/FRt1/assignment_loss‚5.370895778431612 1loss/SRqqt2/detection_loss‚3.1298273196245763 1loss/SRqqt4/detection_loss‚3.6362747244718596 .loss/FBt4/detection_loss‚8.276650659739971 /loss/FBt3/detection_loss‚2.6541313858753033VË!W‹Ñ 1loss/SRqqt2/assignment_loss‚2.857789562103596 0loss/FBt4/assignment_loss‚0.4950204354900052 % _timestamp‚1.7884081143448708e+09 _runtime‚ 724.756926024 1loss/SRqqt3/detection_loss‚0.9581430883859707 trainer/global_step‚1049 /loss/FRt3/assignment_loss‚4.625845318255217 2loss/SRqqt3/assignment_loss‚2.7158921232706383 1loss/SRqqt4/assignment_loss‚3.294554454524343 0loss/FBt1/assignment_loss‚0.3111364713430786 0loss/FBt2/assignment_loss‚0.5042653986149364 1loss/FBt3/assignment_loss‚0.20901517895364316 /loss/FRt4/assignment_loss‚6.324271869659424 2loss/SRqqt1/assignment_loss‚2.8929188777418697 .loss/FRt4/detection_loss‚2.937986143430074 .loss/FBt2/detection_loss‚7.287977516651154 /loss/FRt3/detection_loss‚3.3113643222528957 1loss/SRqqt1/detection_loss‚3.4892006432361464 _step‚83 /loss/FRt1/assignment_loss‚5.370895778431612 1loss/SRqqt2/detection_loss‚3.1298273196245763 1loss/SRqqt4/detection_loss‚3.6362747244718596 .loss/FBt4/detection_loss‚8.276650659739971 /loss/FBt3/detection_loss‚2.6541313858753033 /loss/FBt1/detection_loss‚4.2455385133172525 epoch‚0 .loss/FRt1/detection_loss‚4.774443692144225 /loss/FRt2/detection_loss‚3.0816158509025207 %loss/total_loss‚82.87383816651194 .loss/FRt2/assignment_loss‚5.48907883809163Syš' Œ‚(:… ¼âãÔûÉœ™ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚92.112' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.4224 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pÂÒ‚(: ¼âãÔ³ýÄœ disk.nvme0n1.in‚0.03125 disk.sda3.in‚0 disk.sda.out‚ 31.60546875$ memory_percent‚96.62978649139404( proc.memory.availableMB‚ 552.17578125 disk.dm-1.in‚0# proc.memory.rssMB‚ 2782.69921875 network.sent‚ 1.262684e+06 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚0.03125 disk.sda3.out‚ 31.60546875* proc.memory.percent‚16.984248161315918% disk./.usageGB‚1056.0574913024902 disk.dm-1.out‚ 31.60546875 disk.rbd3.out‚0 disk.nvme0c0n1.in‚0.03125 disk.sda1.in‚0" disk.dm-0.out‚2911.15185546875 disk.rbd2.out‚ 17.0478515625' disk.nvme0c0n1.out‚2911.15185546875 disk.sda.in‚0 network.recv‚338281) disk./.usagePercent‚70.87290433256501 disk.rbd1.out‚ 0.61279296875% disk.nvme0n1.out‚2911.15185546875 cpu‚25.08107233166158 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd3.in‚0¡œý‘Ž‚(:† ËâãÔ¥ÿ™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.102' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6204 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝÐï‚(:Š ËâãÔ¡ú œ disk.rbd0.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-0.in‚0.03125# memory_percent‚96.6282844543457# proc.memory.rssMB‚ 2782.69921875 network.sent‚ 1.311788e+06 disk.dm-1.out‚ 32.21484375 disk.rbd0.out‚0 disk.sda3.in‚0 cpu‚25.08081248273834 disk.dm-1.in‚0 disk.rbd2.out‚ 18.7470703125 disk.nvme0c0n1.in‚0.03125' disk.nvme0c0n1.out‚2911.59033203125 disk.sda3.out‚ 32.21484375 disk.sda.out‚ 32.21484375 disk.nvme0n1.in‚0.03125% disk./.usageGB‚1056.0574913024902 disk.drbd1777.in‚0* proc.memory.percent‚16.984248161315918 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚342863 disk.sda2.in‚0 disk.drbd1777.out‚0" disk.dm-0.out‚2911.59033203125% disk.nvme0n1.out‚2911.59033203125) disk./.usagePercent‚70.87290433256501 disk.sda.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.61279296875& proc.memory.availableMB‚ 552.421875gŠ–š0+ _wandbruntime‚758  _runtime‚758zeàÄ‘¾ & lr-AdamW/pg1‚9.184219838669073e-05 & lr-AdamW/pg2‚9.184219838669073e-05 trainer/global_step‚1099 % _timestamp‚1.7884081479060931e+09 _runtime‚ 758.31824444 _step‚84× Â¯È’Â & lr-AdamW/pg1‚9.184219838669073e-05 & lr-AdamW/pg2‚9.184219838669073e-05 trainer/global_step‚1099 % _timestamp‚1.7884081479060931e+09 _runtime‚ 758.31824444 _step‚84TÈ—!0“+ _wandbruntime‚758  _runtime‚758¨X-Á'”»O N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 _step‚85 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚31541167354880 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6056904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14179810 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚28547360494592 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1527645508054 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 trainer/global_step‚1099 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21833445 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚30008946413568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1532034705408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13933667 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚33374202322432 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1558536982016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 222112256 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209254400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13933665 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233194496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1558746236416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚30008964416512 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31540981118976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1532034705408 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30075006002646 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35767110 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚30008964416512 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6056375 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚28547377533952 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35785652 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21833445 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21851985 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1532202938368 % _timestamp‚1.7884081479068384e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1527810621910 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31815432145920 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14179810 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚28547360494592 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1527645508054 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 trainer/global_step‚1099 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21833445 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚30008946413568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1532034705408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13933667 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚33374202322432 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1558536982016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 222112256 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209254400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13933665 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233194496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1558746236416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚30008964416512 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31540981118976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1532034705408 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30075006002646 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35767110 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚30008964416512 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6056375 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚28547377533952 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35785652 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21833445 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21851985 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1532202938368 % _timestamp‚1.7884081479068384e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1527810621910 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31815432145920 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚30036215778304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1533596089856 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35818124 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35799556 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 222112256 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚30036215778304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35818124 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚31569643621888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30102338349526 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8130281 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13946324 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current*yËF‚ 5041553408 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13946326 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14192002 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8130283 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21871798 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚30036233781248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13946324 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚505 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _step‚86 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30102520502950 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚33404529572864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1560160538624 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35799556 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13946326 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14192507 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚28573320716288 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1533427843584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚503 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224792576 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚31569829871104 trainer/global_step‚1099 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6061721 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚2173ÁRëÏ&˜ÉM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1560160538624 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35799556 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13946326 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14192507 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚28573320716288 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1533427843584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚503 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224792576 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚31569829871104 trainer/global_step‚1099 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6061721 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1533427843584 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6062224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31844345094144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1529199786662 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884081485514135e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31844369034240 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200852480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚30036215778304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1533596089856 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35818124 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35799556 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 222112256 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚30036215778304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35818124 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚31569643621888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30102338349526 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8130281 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13946324 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13946326 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14192002 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8130283 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21871798 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚30036233781248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13946324 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚505 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _step‚86 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30102520502950 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚33404529572864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416V¨ÖÅ0™+ _wandbruntime‚758  _runtime‚758 %× MšÇ _step‚87 2loss/SRqqt3/assignment_loss‚2.4694119715690612 .loss/FRt1/detection_loss‚4.731708253131194 /loss/FRt2/assignment_loss‚4.602202473854532 1loss/SRqqt2/assignment_loss‚2.353519856929779 1loss/SRqqt4/assignment_loss‚3.100268703699112 -loss/FRt4/detection_loss‚4.08859971554383 %loss/total_loss‚75.79392213351265 trainer/global_step‚1099 0loss/FBt2/assignment_loss‚0.6885246164165437 1loss/FBt4/assignment_loss‚0.47876326857312096 epoch‚0 1loss/FBt3/assignment_loss‚0.31563896951060827 .loss/FBt1/detection_loss‚4.753402287761371 .loss/FRt3/detection_loss‚3.215993747432181 /loss/FRt1/assignment_loss‚4.902878810377682 /loss/FRt3/assignment_loss‚3.956476647803124 /loss/FRt4/assignment_loss‚6.080609839895497 0loss/FBt1/assignment_loss‚0.4266356284157761 0loss/SRqqt2/detection_loss‚3.006327199128767 /loss/SRqqt3/detection_loss‚1.24858258899198 .loss/FBt2/detection_loss‚3.520815382401332 /loss/FBt3/detection_loss‚1.4186308792410283 /loss/FRt2/detection_loss‚3.2628634906544978 1loss/SRqqt4/detection_loss‚3.7039376735687255 .loss/FBt4/detection_loss‚6.346313810064679 1loss/SRqqt1/assignment_loss‚2.970478418469429 0loss/SRqqt1/detection_loss‚4.151337900078805 % _timestamp‚1.7884081485532157e+09 _runtime‚ 758.965269979ðUÏQ›Ë .loss/FRt1/detection_loss‚4.731708253131194 /loss/FRt2/assignment_loss‚4.602202473854532 1loss/SRqqt2/assignment_loss‚2.353519856929779 1loss/SRqqt4/assignment_loss‚3.100268703699112 -loss/FRt4/detection_loss‚4.08859971554383 %loss/total_loss‚75.79392213351265 trainer/global_step‚1099 0loss/FBt2/assignment_loss‚0.6885246164165437 1loss/FBt4/assignment_loss‚0.47876326857312096 epoch‚0 1loss/FBt3/assignment_loss‚0.31563896951060827 .loss/FBt1/detection_loss‚4.753402287761371 .loss/FRt3/detection_loss‚3.215993747432181 /loss/FRt1/assignment_loss‚4.902878810377682 /loss/FRt3/assignment_loss‚3.956476647803124 /loss/FRt4/assignment_loss‚6.080609839895497 0loss/FBt1/assignment_loss‚0.4266356284157761 0loss/SRqqt2/detection_loss‚3.006327199128767 /loss/SRqqt3/detection_loss‚1.24858258899198 .loss/FBt2/detection_loss‚3.520815382401332 /loss/FBt3/detection_loss‚1.4186308792410283 /loss/FRt2/detection_loss‚3.2628634906544978 1loss/SRqqt4/detection_loss‚3.7039376735687255 .loss/FBt4/detection_loss‚6.346313810064679 1loss/SRqqt1/assignment_loss‚2.970478418469429 0loss/SRqqt1/detection_loss‚4.151337900078805 % _timestamp‚1.7884081485532157e+09 _runtime‚ 758.965269979 _step‚87 2loss/SRqqt3/assignment_loss‚2.4694119715690612WÈ®'Ìœœ‚(:‘ ÚâãÔÅ´Ø™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.061' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.212200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!Iƒ‚(:… ÚâãÔ¢õ´œ$ disk./.usageGB‚1056.057529449463% disk.nvme0n1.out‚3010.71533203125 disk.rbd2.out‚ 18.7626953125 disk.dm-0.in‚0.03125 disk.nvme0c0n1.in‚0.03125 disk.drbd1777.in‚0 network.recv‚347908" disk.dm-0.out‚3010.71533203125 disk.sda2.in‚0 disk.sda1.in‚0$ memory_percent‚96.62995338439941 cpu‚25.07989917810381 disk.sda3.in‚0' proc.memory.availableMB‚ 552.1484375 disk.dm-1.in‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 2782.69921875 proc.cpu.threads‚17 disk.nvme0n1.in‚0.03125 disk.dm-1.out‚ 32.6796875 disk.rbd3.in‚0 disk.sda3.out‚ 32.6796875 disk.sda1.out‚0 disk.rbd1.in‚0) disk./.usagePercent‚70.87290689264036 disk.rbd0.out‚0 disk.rbd3.out‚0' disk.nvme0c0n1.out‚3010.71533203125 disk.sda.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 0.67578125 network.sent‚ 1.318049e+06 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda.out‚ 32.6796875* proc.memory.percent‚16.984248161315918,™”µœž‚(:‘ éâãԤ⶙ gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.821' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.164199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡Ò¤ Ÿ‚(:ÿ éâãÔ“‚Ìœ network.recv‚353163 disk.rbd2.out‚ 18.8251953125 disk.nvme0c0n1.in‚0.03125' disk.nvme0c0n1.out‚3109.88623046875( proc.memory.availableMB‚ 552.08203125) disk./.usagePercent‚70.87289204420337 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.rbd1.in‚0! proc.memory.rssMB‚ 2782.703125 disk.dm-0.in‚0.03125 disk.dm-1.out‚33.28125 disk.rbd2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.out‚0$ memory_percent‚96.63035869598389 disk.sda.out‚33.28125 network.sent‚ 1.367906e+06% disk./.usageGB‚1056.0573081970215 disk.rbd1.out‚ 0.67578125% disk.nvme0n1.out‚3109.88623046875 disk.rbd0.out‚0* proc.memory.percent‚16.984272003173828 disk.drbd1777.out‚0 disk.nvme0n1.in‚0.03125 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 cpu‚25.07933091812169" disk.dm-0.out‚3109.88623046875 disk.rbd3.out‚0 disk.sda3.out‚33.28125Õñ»0 + _wandbruntime‚792  _runtime‚792ôÝéÄá½ & lr-AdamW/pg1‚9.602064235332817e-05 & lr-AdamW/pg2‚9.602064235332817e-05 trainer/global_step‚1149 # _timestamp‚1.78840818237437e+09 _runtime‚ 792.786571949 _step‚88Enx Ç¢Á & lr-AdamW/pg1‚9.602064235332817e-05 & lr-AdamW/pg2‚9.602064235332817e-05 trainer/global_step‚1149 # _timestamp‚1.78840818237437e+09 _runtime‚ 792.786571949 _step‚88Xzì0 0£+ _wandbruntime‚792  _runtime‚792–ÖŸ²&¤ºO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚31372414650368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14824564 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37392762 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6333362 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _step‚89 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22844687 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6332810 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34890706640896 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14566615 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1601706028032 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14566615 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1597117326462 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚32974306914304 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚32974120678400 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22826147 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚29844536650752 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚31372432653312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33261079557120 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1601874260992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33261103497216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚32974120678400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8491754 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8491756 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚31441636937854 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22826147 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚31372414650368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29844519611392 $ _timestamp‚1.788408182375124e+09 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚31441819091070 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 222112256 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚31372432653312 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peœ‹Áwak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 207157248À|8xÄ'¥¾O XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚31372414650368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14824564 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37392762 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6333362 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _step‚89 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22844687 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6332810 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34890706640896 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14566615 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1601706028032 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14566615 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1597117326462 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚32974306914304 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚32974120678400 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22826147 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚29844536650752 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚31372432653312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33261079557120 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1601874260992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33261103497216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚32974120678400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8491754 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8491756 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚31441636937854 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22826147 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚31372414650368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29844519611392 $ _timestamp‚1.788408182375124e+09 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚31441819091070 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 222112256 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚31372432653312 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 207157248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160Y|90¦+ _wandbruntime‚793  _runtime‚793HÞ(8Ë&§ÅM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37425208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚31469151438158 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚31468969284734 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _runtime‚ 793.432457994 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231084032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22845934 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33289992505344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1630817607168 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22864500 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14837412 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _step‚90 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22864500 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1631024751104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29870479833088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚33002969430528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1598506491006 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33290016445440 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8499133 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6338281 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1603267412480 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37443776 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1603099166208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14579274 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22845934 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1603099166208 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14837939 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6338806 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1149 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚31399684015104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚34920810112512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29870462793728 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14579274 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33002969430528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14579276 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚31399684015104 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34921041196544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 222112256 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360}"$rÏ&¨ÉM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22845934 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1603099166208 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14837939 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6338806 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1149 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚31399684015104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚34920810112512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29870462793728 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14579274 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33002969430528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14579276 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚31399684015104 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34921041196544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 222112256 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37425208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚31469151438158 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚31468969284734 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _runtime‚ 793.432457994 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231084032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22845934 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33289992505344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1630817607168 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22864500 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14837412 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _step‚90 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22864500 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1631024751104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29870479833088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚33002969430528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1598506491006 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33290016445440 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8499133 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6338281 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1603267412480 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37443776 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1603099166208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14579274 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408ZÔ /0©+ _wandbruntime‚793  _runtime‚793|si"RªÌ 0loss/FBt2/assignment_loss‚0.4675078522746584 0loss/FBt4/assignment_loss‚0.5764362055890482 /loss/FRt1/detection_loss‚2.8993157600717887 .loss/FRt4/detection_loss‚3.528376889228821 0loss/SRqqt3/detection_loss‚1.444055903909504 1loss/SRqqt2/assignment_loss‚2.613738977909088 1loss/SRqqt3/assignment_loss‚2.745820203248192 1loss/FBt3/assignment_loss‚0.36241000340133295 %loss/total_loss‚82.35596498445167 epoch‚0 /loss/FRt1/assignment_loss‚5.167009166308811 1loss/SRqqt2/detection_loss‚2.7506005456501788 /loss/FRt4/assignment_loss‚6.130689732233683 1loss/FBt1/assignment_loss‚0.44840042214648723 /loss/FRt3/detection_loss‚3.2966521316104465 /loss/FBt2/detection_loss‚11.143555206911904 trainer/global_step‚1149 % _timestamp‚1.7884081830214262e+09 _step‚91 /loss/FRt2/assignment_loss‚4.267113094329834 /loss/FRt3/assignment_loss‚4.542942815356785 1loss/SRqqt1/detection_loss‚3.3348691817957743 0loss/SRqqt4/detection_loss‚3.677012000977993 /loss/FBt1/detection_loss‚3.6625093108898885 _runtime‚ 793.433466874 /loss/FRt2/detection_loss‚3.1361333394050597 /loss/FBt3/detection_loss‚2.7069742778504544 1loss/SRqqt1/assignment_loss‚2.937472532192866 1loss/SRqqt4/assignment_loss‚3.166270023584366 -loss/FBt4/detection_loss‚7.35009940757471žw¥ª«Ð 0loss/FBt2/assignment_loss‚0.4675078522746584 0loss/FBt4/assignment_loss‚0.5764362055890482 /loss/FRt1/detection_loss‚2.8993157600717887 .loss/FRt4/detection_loss‚3.528376889228821 0loss/SRqqt3/detection_loss‚1.444055903909504 1loss/SRqqt2/assignment_loss‚2.613738977909088 1loss/SRqqt3/assignment_loss‚2.745820203248192 1loss/FBt3/assignment_loss‚0.36241000340133295 %loss/total_loss‚82.35596498445167 epoch‚0 /loss/FRt1/assignment_loss‚5.167009166308811 1loss/SRqqt2/detection_loss‚2.7506005456501788 /loss/FRt4/assignment_loss‚6.130689732233683 1loss/FBt1/assignment_loss‚0.44840042214648723 /loss/FRt3/detection_loss‚3.2966521316104465 /loss/FBt2/detection_loss‚11.143555206911904 trainer/global_step‚1149 % _timestamp‚1.7884081830214262e+09 _step‚91 /loss/FRt2/assignment_loss‚4.267113094329834 /loss/FRt3/assignment_loss‚4.542942815356785 ÌsOU¬1loss/SRqqt1/detection_loss‚3.3348691817957743 0loss/SRqqt4/detection_loss‚3.677012000977993 /loss/FBt1/detection_loss‚3.6625093108898885 _runtime‚ 793.433466874 /loss/FRt2/detection_loss‚3.1361333394050597 /loss/FBt3/detection_loss‚2.7069742778504544 1loss/SRqqt1/assignment_loss‚2.937472532192866 1loss/SRqqt4/assignment_loss‚3.166270023584366 -loss/FBt4/detection_loss‚7.35009940757471[ï2vT‘¬‚(:† øâãÔ¸‡«™ gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.926' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7852 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ص­‚(:ý øâãÔ²ž¯œ disk.dm-0.in‚0.03125" disk.dm-0.out‚3110.19482421875 disk.nvme0n1.in‚0.03125% disk.nvme0n1.out‚3110.19482421875! proc.memory.rssMB‚ 2782.703125 proc.cpu.threads‚17 disk.dm-1.out‚33.65625 disk.nvme0c0n1.in‚0.03125 disk.sda1.in‚0$ memory_percent‚96.63271903991699 cpu‚25.07911391162109 disk.rbd0.out‚0 disk.sda.out‚33.65625 disk.drbd1777.in‚0 network.recv‚357087$ disk./.usageGB‚1056.058334350586 disk.rbd2.in‚0 disk.sda3.out‚33.65625 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd3.out‚0' disk.nvme0c0n1.out‚3110.19482421875 disk.sda3.in‚0 disk.rbd1.in‚0' proc.memory.availableMB‚ 551.6953125) disk./.usagePercent‚70.87296091023008 disk.dm-1.in‚0 disk.rbd2.out‚ 18.8408203125 disk.drbd1777.out‚0* proc.memory.percent‚16.984272003173828 network.sent‚ 1.373452e+06 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 0.67578125ˆfHŽ®‚(:ƒ ‡ããÔ‰ãÑ™ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.75' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.35 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ܨ ¯‚(:û ‡ããÔ˜²›œ disk.sda.out‚34.28125! disk.dm-0.out‚3209.4658203125 disk.dm-1.out‚34.28125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.03125 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚0.03125$ memory_percent‚96.63300514221191! proc.memory.rssMB‚ 2782.703125 network.sent‚ 1.422565e+06( disk./.usagePercent‚70.8729580941472$ disk./.usageGB‚1056.058292388916 disk.rbd1.out‚ 0.67578125* proc.memory.percent‚16.984272003173828 disk.sda2.in‚0 disk.sda3.out‚34.28125 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚0.03125 disk.dm-1.in‚0 proc.cpu.threads‚17& disk.nvme0c0n1.out‚3209.4658203125 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.sda3.in‚0 cpu‚25.078299498955598 disk.rbd0.in‚0 disk.rbd2.out‚18.99365234375 disk.rbd3.in‚0$ disk.nvme0n1.out‚3209.4658203125' proc.memory.availableMB‚ 551.6484375 network.recv‚361829 disk.rbd2.in‚0 disk.rbd3.out‚0WQÒ‘°‚(:† –ããÔµÖ÷š gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚101.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.3678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎQ%,±‚(:… –ããÔâÏ– network.sent‚ 1.428837e+06 disk.nvme0c0n1.in‚0.03125 disk.sda.out‚ 34.5859375 disk.drbd1777.in‚0 disk.rbd2.out‚19.00927734375$ disk./.usageGB‚1056.058292388916 disk.rbd3.out‚0 disk.rbd0.out‚0 network.recv‚366995 disk.sda3.out‚ 34.5859375 disk.nvme0n1.in‚0.03125 disk.dm-1.in‚0 disk.dm-1.out‚ 34.5859375" disk.dm-0.out‚3308.59716796875( disk./.usagePercent‚70.8729580941472 disk.sda2.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚3308.59716796875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚16.984272003173828$ memory_percent‚96.63288593292236 cpu‚25.077741567903697 disk.rbd3.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚0.03125 disk.rbd2.in‚0! proc.memory.rssMB‚ 2782.703125' disk.nvme0c0n1.out‚3308.59716796875 disk.sda3.in‚0 disk.rbd1.out‚ 0.67578125( proc.memory.availableMB‚ 551.66796875Q ¦0²+ _wandbruntime‚826  _runtime‚826ê³;Vų¿ % _timestamp‚1.7884082164587367e+09 _runtime‚ 826.870849745 _step‚92 & lr-AdamW/pg1‚0.0001001990863199656 & lr-AdamW/pg2‚0.0001001990863199656 trainer/global_step‚1199ü •;É´Ã & lr-AdamW/pg2‚0.0001001990863199656 trainer/global_step‚1199 % _timestamp‚1.7884082164587367e+09 _runtime‚ 826.870849745 _step‚92 & lr-AdamW/pg1‚0.0001001990863199656\+;¸U0µ+ _wandbruntime‚826  _runtime‚826©ï]2Á'¶»O _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚32735900890112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1666589144870 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8860604 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224805888 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1671545583616 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15470907 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15470393 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚31141678728192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34407260237824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1671545583616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15199565 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚ 826.871779456 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34706726968320 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚514 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6610301 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 222112256 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15199565 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚34407446473728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚32735882887168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39036956 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚36407216965632 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23818849 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚34407446473728 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1671545583616 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15470907 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15470393 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚31141678728192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34407260237824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1671545583616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15199565 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚ 826.871779456 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34706726968320 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚514 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6610301 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 222112256 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15199565 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚34407446473728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚32735882887168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39036956 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160]ÿü›0¸+ _wandbruntime‚827  _runtime‚827ZGDy¹ÃM L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1672938735104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1667978309414 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _step‚94 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚34435922740736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39069428 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1701680965120 trainer/global_step‚1199 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23857202 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39050860 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 207143936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚32763170254848 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15212224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231084032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23857202 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39050860 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚32763152251904 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15483122 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚32763152251904 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15212226 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1672770488832 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6615653 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚36437320881664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚32835600219942 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚36437551965696 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1701888109056 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_ba²»6sP tch_end/requested_bytes.small_pool.peak‚ 406782421 _runtime‚ 827.5153766 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1672770488832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23838636 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚34436108989952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚32835782373366 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1672938735104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15212224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23838636 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8867983 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚31167638949888 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚31167621910528 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39069428 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15483636 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1668143423478hÌèbÍ&ºÇM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 _runtime‚ 827.5153766 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1672770488832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23838636 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚34436108989952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚32835782373366 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1672938735104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15212224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23838636 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8867983 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚31167638949888 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚31167621910528 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39069428 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15483636 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1668143423478 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1672938735104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1667978309414 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _step‚94 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚34435922740736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39069428 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1701680965120 trainer/global_step‚1199 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23857202 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39050860 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 207143936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚32763170254848 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15212224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231084032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23857202 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39050860 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚32763152251904 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15483122 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚32763152251904 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15212226 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1672770488832 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6615653 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚36437320881664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚32835600219942 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚36437551965696 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1701888109056 B¿‚(:… ¥ããÔ§—ú› disk.nvme0c0n1.in‚0.03125 disk.sda1.in‚0 disk.sda1.out‚0" disk.dm-0.out‚3309.32275390625 disk.rbd2.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2858.59765625 disk.sda2.out‚0% disk.nvme0n1.out‚3309.32275390625 network.recv‚371467 disk.sda.out‚ 35.203125 disk.rbd1.out‚ 0.67578125 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 35.203125( proc.memory.availableMB‚ 475.32421875' disk.nvme0c0n1.out‚3309.32275390625 disk.sda.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.nvme0n1.in‚0.03125 network.sent‚ 1.435012e+06) disk./.usagePercent‚70.87296142224515% disk./.usageGB‚1056.0583419799805 disk.dm-1.in‚0 disk.dm-1.out‚ 35.203125 disk.rbd0.in‚0 disk.rbd2.out‚20.64599609375 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚0.03125 disk.sda2.in‚0$ memory_percent‚97.09885120391846* proc.memory.percent‚17.447495460510254 cpu‚25.11318715663338 proc.cpu.threads‚17Ûù'ŽÀ‚(:ƒ ´ããÔ ŠÚ› gpu.0.gpu‚44.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.34' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z½~\Á‚(:û ´ããÔü‚¼ž disk.sda1.out‚0( proc.memory.availableMB‚ 474.94921875* proc.memory.percent‚17.453908920288086) disk./.usagePercent‚70.87296142224515' disk.nvme0c0n1.out‚3408.41650390625 disk.sda2.out‚0 disk.sda3.in‚0 cpu‚25.19637065516945 network.recv‚376515 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.03125 disk.sda.in‚0 proc.cpu.threads‚17 network.sent‚ 1.484015e+06 disk.rbd2.out‚20.66162109375 disk.rbd1.in‚0% disk.nvme0n1.out‚3408.41650390625 disk.rbd0.out‚0 disk.nvme0n1.in‚0.03125 disk.drbd1777.in‚0% disk./.usageGB‚1056.0583419799805 disk.dm-1.in‚0 disk.dm-1.out‚35.625 disk.sda1.in‚0 disk.dm-0.in‚0.03125" disk.dm-0.out‚3408.41650390625 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda.out‚35.625 disk.drbd1777.out‚0$ memory_percent‚97.10114002227783" proc.memory.rssMB‚ 2859.6484375 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 0.67578125 disk.sda3.out‚35.625²OBé›Â‚(: ÃããÔýù”™ gpu.0.gpu‚43.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.237' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.647399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù‡aE Â(:ÿ ÃããÔ˜¬õœ network.sent‚ 1.489411e+06) disk./.usagePercent‚70.87296167825268 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.rbd0.out‚0$ memory_percent‚97.10016250610352* proc.memory.percent‚17.452216148376465 disk.sda1.in‚0 disk.dm-0.out‚3507.505859375 disk.rbd3.in‚0% disk./.usageGB‚1056.0583457946777 disk.sda1.out‚0 disk.dm-1.out‚ 36.2109375% disk.nvme0c0n1.out‚3507.505859375 network.recv‚380357# disk.nvme0n1.out‚3507.505859375 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 0.67578125 disk.rbd2.out‚ 20.751953125 disk.nvme0c0n1.in‚0.03125 disk.sda2.in‚0 disk.sda3.out‚ 36.2109375 disk.nvme0n1.in‚0.03125 disk.sda2.out‚0& proc.memory.availableMB‚ 475.109375# proc.memory.rssMB‚ 2859.37109375 cpu‚25.277841736837548 disk.sda.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚0.03125 disk.dm-1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 36.2109375 disk.drbd1777.in‚0uR[¾iÄbd ADeviceStatsMonitor.on_validation_batch_start/active.all.allocated"trainer/global_step2:JPîÀä{tÅbo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPEàhÆbc @DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak"trainer/global_step2:JPl‡Ý²tÇbo LDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JP“z@ÄoÈbj GDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current"trainer/global_step2:JPÃ;þoÉbj GDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current"trainer/global_step2:JP¼é]tÊbo LDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JPBŒÏûmËbh EDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak"trainer/global_step2:JP5Þ®vÌbq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JPá:'tÍbo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JP…â¼çoÎbj GDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak"trainer/global_step2:JPõ uÏbp MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JP'CfrÐbm JDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPT®”-oÑbj GDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed"trainer/global_step2:JPýõ¤zyÒbt QDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JP‘‡"kÓbf CDeviceStatsMonitor.on_validation_batch_start/allocation.all.current"trainer/global_step2:JPwx„lÔbg DDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed"trainer/global_step2:JPªé5rÕbm JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP™ò\cvÖbq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPéò#Ám×bh EDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated"trainer/global_step2:JPӼϤnØbi FDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak"trainer/global_step2:JPáñjÌuÙbp MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JP¸£qtÚbo LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JPôçiòrÛbm JDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current"trainer/global_step2:JPÜö­Ö~Üby VDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPõà]üeÝb` =DeviceStatsMonitor.on_validation_batch_start/segment.all.peak"trainer/global_step2:JP¦†ù^qÞbl IDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JPµmßbh EDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak"trainer/global_step2:JPvœó†vàbq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPëåM\mábh EDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed"trainer/global_step2:JP¨ÖD7lâbg DDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak"trainer/global_step2:JPGºá×|ãbw TDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JPó^ä‡täbo LDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JPdÄsåbn KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JP³­Ò¡oæbj GDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current"trainer/global_step2:JP>sŒ-kçbf CDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak"trainer/global_step2:JPZK Ítèbo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPÁåbtébo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JPÜ"¯]êbX 5DeviceStatsMonitor.on_validation_batch_start/num_ooms"trainer/global_step2:JPId}Álëbg DDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed"trainer/global_step2:JP—Ùä\pìbk HDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed"trainer/global_step2:JPö“Ñþpíbk HDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated"trainer/global_step2:JP èÆztîbo LDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JP†+„sïbn KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JP~$qðbl IDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated"trainer/global_step2:JP‡@±Ì~ñby VDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPIç‚åeòb` =DeviceStatsMonitor.on_validation_batch_start/num_device_alloc"trainer/global_step2:JPæw©‚qóbl IDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated"trainer/global_step2:JPEidôb_ ¼xšbs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JP~{Òs›bn KDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated"trainer/global_step2:JPïøÝlœbg DDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak"trainer/global_step2:JP;ZKksbn KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPgÝGtkžbf CDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak"trainer/global_step2:JP)j2†wŸbr ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JP±GÍùq bl IDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JPO©gr¡bm JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JP}1‹Âl¢bg DDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed"trainer/global_step2:JP·ß}=f£ba >DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries"trainer/global_step2:JPT8Þyn¤bi FDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current"trainer/global_step2:JP8Ðh¥bc @DeviceStatsMonitor.on_validation_batch_start/segment.all.current"trainer/global_step2:JP.Ò¨v¦bq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JP©<•’x§bs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JP¶'«˜|¨bw TDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JPòÊóix©bs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JP2µ(yªbt QDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JP.5£v«bq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current"trainer/global_step2:JPÅ lëc¬b^ ;DeviceStatsMonitor.on_validation_batch_start/max_split_size"trainer/global_step2:JP0šûli­bd ADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams"trainer/global_step2:JPÊù¥E®ba >DeviceStatsMonitor.on_validation_batch_start/segment.all.freedÔ\ßð!"trainer/global_step2:JP>o¯bj GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated"trainer/global_step2:JPŽÍn°bi FDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed"trainer/global_step2:JP+ð0Bn±bi FDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed"trainer/global_step2:JPÆ•w²br ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JPÆœ*ám³bh EDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed"trainer/global_step2:JPí^Æn´bi FDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current"trainer/global_step2:JPGßÃírµbm JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JP÷¡ã[u¶bp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JP¨2Ѷd·b_ DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 1702599595520 \PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚9252876 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 103780864 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚33932988641280 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ZGDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚35635588236800 YFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚35635443328000 YFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚33975857388038.¢ ,*Á¦T QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 4911254016 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 144908800 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2460 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 1731698138624 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚16163853 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 5041553408 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2460 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚33932970638336 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚33932970638336 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚26 \PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚6910977 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚217 ^KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚37688129302016 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 123922316 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 5041553408 ]JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚35635588236800 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚24382451 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 1702599595520 \PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚9252876 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 103780864 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚33932988641280 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ZGDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚35635588236800 YFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚35635443328000 YFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚33975857388038 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884082697575345e+09 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15191 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 1702472689664 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 1702472689664 Bs„wÜbr ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPfŸöérÝbm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JP£ŸýÍrÞbm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current"trainer/global_step2:JPI.nßbi FDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current"trainer/global_step2:JPèú|sàbn KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JPÌÑÜ sábn KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JP:!EÊrâbm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JP€ããkãbf CDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak"trainer/global_step2:JP°~øYqäbl IDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JPò;úXqåbl IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JP›u+;wæbr ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPòÌc eçb` =DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections"trainer/global_step2:JP›²œucèb^ ;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak"trainer/global_step2:JP5Ùînébi FDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed"trainer/global_step2:JP°ò•mêbh EDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak"trainer/global_step2:JP²²@Cgëbb ?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams"trainer/global_step2:JP E\Qnìbi FDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated"trainer/global_step2:JPFQqjíbe BDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak"trainer/global_step2:JPJ»­§oîbj GDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPPÁ\mïbh EDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current"trainer/global_step2:JPèŘ|ðbw TDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JP²óD|ñbw TDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP´·Œuòbp MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JP¤ÜçÓpóbk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JP @FÒuôbp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JPƨîbõb] :DeviceStatsMonitor.on_validation_batch_end/num_device_free"trainer/global_step2:JP¡¨röbm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JP#B–i÷bd ADeviceStatsMonitor.on_validation_batch_end/allocation.all.current"trainer/global_step2:JPänøbi FDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated"trainer/global_step2:JPÚÝáÚuùbp MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JPÛ3›múbh EDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current"trainer/global_step2:JPûÝŒølûbg DDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current"trainer/global_step2:JP¤9Îàkübf CDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current"trainer/global_step2:JPPB8árýbm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPò.ßÎvþbq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPdÕckÿbf CDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed"trainer/global_step2:JPn3ïÜc€b^ ;DeviceStatsMonitor.on_validation_batch_end/active.all.freed"trainer/global_step2:JP3*JÈibd ADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak"trainer/global_step2:JP î'i‚bd ADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak"trainer/global_step2:JP ê³tƒbo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JPÁƒYov„bq NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JP¨B1\c…b^ ;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc"trainer/global_step2:JPA]ñïm†bh EDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed"trainer/global_step2:JPOx2o‡bj GDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated"trainer/global_step2:JPÚ Þ÷tˆbo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JPBøzÜ[‰bV 3DeviceStatsMonitor.on_validation_batch_end/num_ooms"trainer/global_step2:JPm„kjŠbe BDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak"trainer/global_step2:JP~žùp‹bk HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JP5£ê rŒbm JDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated"trainer/global_step2:JP>³0qbl IDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JPÞN‹ÀlŽbg DDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current"trainer/global_step2:JPºãCürbm JDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated"trainer/global_step2:JPæq÷ 3bj GDeviceStatsMonitor.on_validation_batch_end/oÚ¯Ûÿ<versize_allocations.current"trainer/global_step2:JPßÊ%”l‘bg DDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak"trainer/global_step2:JPÿXçp’bk HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated"trainer/global_step2:JPàËÓk“bf CDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed"trainer/global_step2:JP[CUk”bf CDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed"trainer/global_step2:JPl«0Ew•br ODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JPÓBÚq–bl IDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated"trainer/global_step2:JPJTÜši—bd ADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak"trainer/global_step2:JP¨Aç€r˜bm JDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JPås™bn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JP±ƒŠKtšbo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JPê”e›b` =DeviceStatsMonitor.on_validation_batch_end/active.all.current"trainer/global_step2:JPí:$mœbh EDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated"trainer/global_step2:JP,ßÃiobj GDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JP—Jfžba >DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak"trainer/global_step2:JPj!†mŸbh EDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak"trainer/global_step2:JPúÏpðv bq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JP¾K ßu¡bp MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JPØŒtp¢bk HDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current"trainer/global_step2:JP¶Oºp£bk HDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current"trainer/global_step2:JPŒ¬Cz¤bu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JP᫶p¥bk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JPq`Žw¦br ODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JP"Ò{ x§bs PDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPÞ pt¨bo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JPžid>j©be BDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak"trainer/global_step2:JP`©‹ªwªbr ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JP)¿Ø—r«bm JDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPð³Rk¬bf CDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPnrJn­bi FDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed"trainer/global_step2:JPåÃo®bj GDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated"trainer/global_step2:JP½;Éj¯be BDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak"trainer/global_step2:JPD$üs°bn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JP©Svl±bg DDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed"trainer/global_step2:JPc˃t²bo LDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current"trainer/global_step2:JPisjr³bm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPá56z´bu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JP[µQ!nµbi FDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current"trainer/global_step2:JPîp1f¶ba >DeviceStatsMonitor.on_validation_batch_end/segment.all.current"trainer/global_step2:JPƒö‘Hj·be BDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed"trainer/global_step2:JPk‹ r¸bm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JPÌ%f8q¹bl IDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JPPeSUoºbj GDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated"trainer/global_step2:JP6épíl»bg DDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed"trainer/global_step2:JPPä4q¼bl IDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JPû ¡.0½+ _wandbruntime‚881  _runtime‚881¥"éý1)¾«R L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚40251602 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 127720960 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚40266795 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚283 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚22903 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚22630 ZNDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚9259376 `MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚37708209724416 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15191 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 103780864 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 1702821825536 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 1702948731392 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 1732047274496 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚217 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 1698042235602 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 4911254016 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚24389847 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚16175446 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1003 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 1697918313286 WDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚35654657649664 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 VJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚9259374 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 140961676 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 $ _timestamp‚1.788408270717443e+09 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚24389847 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 406782421 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚199 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚33994106548038 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 417333248 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚217 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 4911254016 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 126905856 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚32296205274112 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚22903 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 126905856 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚40266795 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚16175161 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚22903 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚22630 ZNDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚9259376 `MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚37708209724416 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15191 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 103780864 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 1702821825536 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 1702948731392 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 1732047274496 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚217 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 1698042235602 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 4911254016 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚24389847 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚16175446 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1003 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 1697918313286 WDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚35654657649664 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 VJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚9259374 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 140961676 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 $ _timestamp‚1.788408270717443e+09 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚24389847 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 406782421 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚199 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚33994106548038 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 417333248 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚217 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 4911254016 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 126905856 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚32296205274112 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248a¡$ÑÀ‚(:… ÒããÔÆò­™ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚120.799' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1598 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü^S»Á‚(:÷ ÒããÔ«ËÕœ disk.rbd0.in‚0 disk.sda3.in‚0 network.recv‚385303) disk./.usagePercent‚70.87296167825268 disk.rbd3.in‚0 network.sent‚ 1.495757e+06 disk.dm-1.out‚36.6875 disk.rbd2.in‚0 disk.nvme0c0n1.in‚0.03125 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1056.0583457946777 disk.dm-0.in‚0.03125 disk.rbd3.out‚0 disk.sda1.in‚0# disk.nvme0n1.out‚3507.947265625% disk.nvme0c0n1.out‚3507.947265625 disk.sda3.out‚36.6875 disk.sda.out‚36.6875 disk.nvme0n1.in‚0.03125$ memory_percent‚97.20284938812256) proc.memory.percent‚17.55225658416748 disk.rbd2.out‚ 20.767578125 disk.sda2.out‚0 disk.rbd1.out‚ 0.67578125( proc.memory.availableMB‚ 458.28515625# proc.memory.rssMB‚ 2875.76171875 cpu‚25.358749228749176 disk.dm-0.out‚3507.947265625 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda1.out‚0ŸºsÈJÂbE "Purity/3FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPÚKQý@Ãb; particle/accuracy_5_of_6"trainer/global_step2:JPØÿ7IHÄbC Purity/2FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JP·qÔJÅbE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP­pùŒHÆbC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPg0ÌéNÇbI &Purity/3FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPx/7ê;Èb6 jet/accuracy_5_of_5"trainer/global_step2:JPðUUJHÉbC Purity/2FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP¹ÿ¦“JÊbE "Purity/*FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP‡UJËbE "Purity/0FRt2SRqqt3FBt/event_purity"trainer/global_step2:JPÛ{eJÌbE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPÂלNÍbI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPÛëÁUJÎbE "Purity/3FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP3Q’þHÏbC Purity/3FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPw÷QiHÐbC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPí7‘NÑbI &Purity/*FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPŠãBNÒbI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP%SîJÓbE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP-¢°tJÔbE "Purity/1FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP$ª¹MNÕbI &Purity/3FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP"ÏNÖbI &Purity/4FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPRpUH×bC Purity/3FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP׃„HØbC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP›È»NÙbI &Purity/2FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP¤ƒ~HÚbC Purity/3FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPôhNÛbI &Purity/3FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPê ÖJÜbE "Purity/*FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPb<…HHÝbC Purity/*FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP%Ü¥8HÞbC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP§_'JßbE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPjÍ°LJàbE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP‰Ê¡«@áb; particle/accuracy_1_of_6"trainer/global_step2:JPÑ«ªNâbI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP«sÊNãbI &Purity/1FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP¬ÓékHäbC Purity/3FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPWJHåbC Purity/1FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JP#ä!kHæbC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPôÖ—HçbC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPòDÅ5NèbI &Purity/2FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPôÆJNébI &Purity/3FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPEÓ«òHêbC Purity/3FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPé_T1@ëb; particle/accuracy_2_of_6"trainer/global_step2:JP+Æ™J¤ bE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP|§úÜN¥ bI &Purity/3FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPlÔN¦ bI &Purity/4FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP5§ûƒ@§ b; particle/accuracy_5_of_5"trainer/global_step2:JP©ãÙÛJ¨ bE "Purity/*FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPNø‘4J© bE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPeg®3Hª bC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPíë^íH« bC Purity/2FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPÆ@J¬ bE "Purity/0FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP#³«ÕN­ bI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPÙí.H® bC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPíE.N¯ bI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPøõFH° bC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP w6¢H± bC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP»pTN² bI &Purity/2FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP{·q—N³ bI &Purity/2FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP~Ão@´ b; particle/accuracy_2_of_4"trainer/global_step2:JP”ãaHµ bC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP[ ÄcN¶ bI &Purity/2FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP^7û$J· bE "Purity/3FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP%Û>J¸ bE "Purity/0FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPtœ$ºJ¹ bE "Purity/2FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPê^–;º b6 jet/accuracy_6_of_7"trainer/global_step2:JPD9?tJ» bE "Purity/1FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP[ïŽQH¼ bC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP‘I"\H½ bC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP¥·N¾ bI &Purity/0FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPÎp'ãJ¿ bE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÍ)ÐäNÀ bI &Purity/4FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP !¨HÁ bC Purity/1FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP'(p`@ b; particle/accuracy_3_of_6"trainer/global_step2:JPiN>Nà bI &Purity/1FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPg=ïÕNÄ bI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPd¦¼!NÅ bI &Purity/4FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPûÅ7MGÆ bB validation_average_jet_accuracy"trainer/global_step2:JP‚JâJÇ bE "Purity/0FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPJQnJÈ bE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¶à‘HÉ bC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPWø¶JÊ bE "Purity/3FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPçîHË bC Purity/0FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JPÊ H@Ì b; particle/accuracy_2_of_3"trainer/global_step2:JPÙp JÍ bE "Purity/1FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPá›RÀNÎ bI &Purity/*FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPr\¤+HÏ bC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPâ×mJÐ bE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP(‹ÀÓ@Ñ b; particle/accuracy_3_of_4"trainer/global_step2:JPþ8=ÎHÒ bC Purity/1FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPÍDMOJÓ bE "Purity/4FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPŽŸ­´JÔ bE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP „®¨HÕ bC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPJ3hNÖ bI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP@—ÑjN× bI &Purity/3FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPÂÿÍJØ bE "Purity/0FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPÞë HÙ bC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPÑüNÚ bI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP®Ò©HÛ bC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPÿ/nANÜ bI &Purity/2FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP1‘ÇŒHÝ bC Purity/0FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JP0šyJÞ bE "Purity/3FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPäWHÝJß bE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPû1LNà bI &Purity/*FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP¼@½¨Ná bI &Purity/2FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPÿÓ¤!Nâ bI &Purity/4FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPI<™ @ã b; particle/accuracy_1_of_3"trainer/global_step2:JPN9âø@ä b; particle/accuracy_3_of_3"trainer/global_step2:JP‘ê’ŸHå bC Purity/2FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP 0/Næ bI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPP©ÍHç bC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP;s#Nè bI &Purity/*FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP|{FJé bE "Purity/*FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPöRENê bI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP,TzôNë bI &Purity/3FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPðc¹iNì bI &Purity/3FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP{uJí bE "Purity/3FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP[@Hî bC Purity/3FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP.îu²Hï bC Purity/4FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPã3—Nð bI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP]’jHñ bC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPuV žNò bI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPj¯¸Jó bE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPBy¥Jô bE "Purity/2FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPS€dHõ bC Purity/3FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP>/Ò\Jö bE "Purity/3FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP=•&LN÷ bI &Purity/4FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP8¯œJø bE "Purity/*FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPãºaŠHù bC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP6y÷äJú bE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPuö]jJû bE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPó™ÎÌHü bC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPIQ Jý bE "Purity/1FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPòëü"Jþ bE "Purity/0FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP.ã\=Hÿ bC Purity/3FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP†Q^WJ€ bE "Purity/3FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPvP“‹H bC Purity/*FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPöù¼ÇJ‚ bE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPáýÎÛHƒ bC Purity/*FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JP0ÝòN„ bI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPO>x N… bI &Purity/0FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP¥n$H† bC Purity/1FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPŠÜTJ‡ bE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÌÕ-ÑNˆ bI &Purity/2FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPEýdN‰ bI &Purity/3FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPy4ŠÎNŠ bI &Purity/3FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPGþY`@‹ b; particle/accuracy_2_of_5"trainer/global_step2:JPÉiv¥NŒ bI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPæÅÒN bI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPDœò®NŽ bI &Purity/1FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP•étN bI &Purity/3FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP†Í *@ b; particle/accuracy_4_of_4"trainer/global_step2:JPH©®¾0‘ + _wandbruntime‚893  _runtime‚893/›XÓw0’ à¡ >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.06060606241226196 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 & Purity/*FRt*SRqqt3FBt/FRt_purity‚0 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976280152797699 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.2541034519672394 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 jet/accuracy_7_of_7‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.24866147339344025 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.8255360722541809 particle/accuracy_5_of_7‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9862884283065796 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2812499701976776 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.2450396716594696 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.09555555135011673 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.2608680725097656 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.25 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9952381253242493 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.18000000715255737 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/1FRt0SRqqt3FBt/FRt_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.11763393133878708 +jet/accuracy_2_of_6‚0.14743590354919434 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 )loss/val_total_loss‚72.33659098831298 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_4‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.0754108726978302 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 +jet/accuracy_3_of_4‚0.01792186312377453 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.973958432674408 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.14541487395763397 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.1842261701822281 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.002221800619736314 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.09602027386426926 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.0800505056977272 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0853489488363266 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.45614030957221985 particle/accuracy_6_of_6‚1 trainer/global_step‚1208 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.06417357176542282 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.989780843257904 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_1_of_7‚0.5 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.0983884185552597 <"Purity/0FRt2SRqqt*FBt/event_purity‚0.0054921116679906845 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9955146312713623 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9992781281471252 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.15885384380817413 6 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.055555559694767 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.10000000149011612 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.0833333358168602 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.06382978707551956 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.22470393776893616 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.30092594027519226 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.3571428656578064 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.2571732699871063 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.1041666641831398 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9968253970146179 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.20402298867702484 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.2991184890270233 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9924498796463013 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ) Purity/1FRt3SRqqt1FBt/FRt_purity‚0.25 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.07356321066617966 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 -jet/accuracy_3_of_3‚0.0021989066153764725 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2627944052219391 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.16000762581825256 & Purity/1FRt*SRqqt3FBt/FRt_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 _runtime‚ 893.519104362 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.10422709584236145 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.020951736718416214 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9952903985977173 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.13953489065170288 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.43939390778541565 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.2753693759441376 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.29554590582847595 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_4‚0 (particle/accuracy‚0.7617033123970032 particle/accuracy_1_of_7‚1 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.12917324900627136 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.4545454680919647 jet/accuracy_4_of_5‚0 +particle/sensitivity‚0.4167248010635376 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.20370370149612427 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9338061809539795 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.13763228058815002 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.09389979392290115 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.1270810067653656 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.006998557597398758 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9875147342681885 ("Purity/2FRt*SRqqt*FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.0807640329003334 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.08181817829608917 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_3_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.07167655974626541 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9887213706970215 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.04891591519117355 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0003779289545491338 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.10548298060894012 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.07202756404876709 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9907243251800537 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.1680491715669632 7 Purity/*FRtÒ4‘p`1SRqqt1FBt/FBt_purity‚0.9952690601348877 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1585247963666916 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.07286413013935089 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.0714285746216774 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9978147149085999 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.04891591519117355 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10517241060733795 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.09841582179069519 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.0625 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.10000000149011612 jet/accuracy_6_of_7‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.1248234286904335 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9918533563613892 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.13151361048221588 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 6validation_average_jet_accuracy‚0.2743946611881256 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2682923972606659 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.10850971192121506 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.25 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.13119292259216309 particle/accuracy_3_of_4‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.17140379548072815 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09817462414503098 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08392007648944855 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.13201744854450226 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.06360699981451035 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_3‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.10544366389513016 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.20061728358268738 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.13107596337795258 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.13240858912467957 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.1899999976158142 % _timestamp‚1.7884082831047134e+09 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.1291632205247879 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.14856433868408203 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.002785799792036414 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.01024590153247118 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.08681917190551758 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26517078280448914 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.07514619827270508 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.12074830383062363 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_2_of_5‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_4_of_4‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_6‚1 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.12486596405506134 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9926981925964355 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 jet/accuracy_5_of_5‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9976012706756592 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.24923323094844818 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9691091775894165 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 9 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.061302684247493744 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.125 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9988662600517273 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.19913794100284576 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_6‚1 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09138651937246323 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9914901852607727 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08531393110752106 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.2483602911233902 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_4‚0.15426862239837646 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.12117090821266174 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09842006862163544 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.2816145420074463 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.10555555671453476 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9210527539253235 particle/accuracy_4_of_6‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.1313561350107193 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.18891827762126923 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9832661151885986 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.3499999940395355 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.0833333358168602 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.6963561773300171 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.10000000149011612 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.13344024121761322 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.0426839217543602 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.12684223055839539 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09741523861885071 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.601687490940094 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.0756378173828125 + Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.1582465022802353 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.04583093151450157 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.34621378779411316 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.040781717747449875 ;"Purity/0FRt1SRqqt1FBt/event_purity‚0.059624847024679184 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.0763888880610466 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 _step‚98 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.08530998975038528 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.09091856330633163 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.11544959992170334 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.12486596405506134 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.02045890875160694 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.06230928376317024 *jet/accuracy_1_of_1‚0.2900933027267456 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 particle/accuracy_2_of_2‚1 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt3FBt/FRt_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.0453655906021595 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.0534752756357193 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.20000000298023224 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.1979166716337204 * Purity/3FRt1SRqqt*FBt/FRt_purity‚0.125 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8157181739807129 particle/accuracy_7_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.21810057759284973 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 *jet/accuracy_1_of_3‚0.6380931735038757 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.0833333358168602 epoch‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2977489233016968 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6061235666275024 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.12524880468845367 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.13849402964115143 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.24302132427692413 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.04412408545613289 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.14275267720222473 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_3‚0.09893397986888885 'particle/f_score‚0.3530544936656952 9 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.059259261935949326 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9971330761909485 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.2027938961982727 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.0902777835726738 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 particle/accuracy_4_of_5‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0555555559694767 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.0763888880610466 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.01923076994717121 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.1964285671710968 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.05779558792710304 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9915229678153992 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.17910632491111755 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.1458333283662796 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.12174329161643982 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.3125 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.09783609211444855 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_2_of_7‚1 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/2FRt0SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9949217438697815 ("Purity/1FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.03374934569001198 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.15255288779735565 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896174669265747 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.14954650402069092 jet/accuracy_4_of_6‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.11580733954906464 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.004692082293331623 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.17490054666996002 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.09565415233373642 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.0833333358168602 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9886640310287476 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.12934045493602753 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.15826356410980225 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08294280618429184 +jet/accuracy_2_of_2‚0.04745442420244217 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.33707699179649353 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9861221313476562 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0201774463057518 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.07519309967756271 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.07027027010917664 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.12452448159456253 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_5‚0.018010752275586128 & Purity/2FRt3SRqqt1FBt/FRt_purity‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.34732145071029663 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.0863095223903656 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.10225225239992142 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 particle/accuracy_6_of_7‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.2857142984867096 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_7‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.15017448365688324 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.08729448914527893 *jet/accuracy_1_of_2‚0.5247015357017517 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ;"Purity/*FRt1SRqqt1FBt/event_purity‚0.031236568465828896 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.1099415123462677 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.28154757618904114 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9888883233070374 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.7535586357116699 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0714285746216774 9 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.047774966806173325 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 <"Purity/1FRt1SRqqt1FBt/event_purity‚0.0022675737272948027 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.12121211737394333 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.13582156598567963 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.1469298154115677 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0555555559694767 *jet/accuracy_1_of_6‚0.7692307829856873 jet/accuracy_5_of_7‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.1666666716337204 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638uÌsW‚“ ä¡ & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ;"Purity/*FRt1SRqqt1FBt/event_purity‚0.031236568465828896 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.1099415123462677 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.28154757618904114 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9888883233070374 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.7535586357116699 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0714285746216774 9 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.047774966806173325 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 <"Purity/1FRt1SRqqt1FBt/event_purity‚0.0022675737272948027 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.12121211737394333 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.13582156598567963 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.1469298154115677 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0555555559694767 *jet/accuracy_1_of_6‚0.7692307829856873 jet/accuracy_5_of_7‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.1666666716337204 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.06060606241226196 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 & Purity/*FRt*SRqqt3FBt/FRt_purity‚0 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.976280152797699 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.2541034519672394 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 jet/accuracy_7_of_7‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.24866147339344025 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.8255360722541809 particle/accuracy_5_of_7‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9862884283065796 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2812499701976776 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.2450396716594696 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.09555555135011673 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.2608680725097656 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.25 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9952381253242493 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.18000000715255737 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/1FRt0SRqqt3FBt/FRt_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.11763393133878708 +jet/accuracy_2_of_6‚0.14743590354919434 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 )loss/val_total_loss‚72.33659098831298 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_4‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.0754108726978302 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 +jet/accuracy_3_of_4‚0.01792186312377453 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.973958432674408 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.14541487395763397 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.1842261701822281 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.002221800619736314 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.09602027386426926 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.0800505056977272 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0853489488363266 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.45614030957221985 particle/accuracy_6_of_6‚1 trainer/global_step‚1208 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 A&Purity/4FRt*SRqqt0FBt/event_pro§¦ë'iqportion‚0.00018601190822664648 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.06417357176542282 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.989780843257904 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_1_of_7‚0.5 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.0983884185552597 <"Purity/0FRt2SRqqt*FBt/event_purity‚0.0054921116679906845 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9955146312713623 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9992781281471252 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.15885384380817413 6 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.055555559694767 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.10000000149011612 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.0833333358168602 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.06382978707551956 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.22470393776893616 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.30092594027519226 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.3571428656578064 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.2571732699871063 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.1041666641831398 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9968253970146179 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.20402298867702484 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.2991184890270233 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9924498796463013 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ) Purity/1FRt3SRqqt1FBt/FRt_purity‚0.25 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.07356321066617966 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 -jet/accuracy_3_of_3‚0.0021989066153764725 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2627944052219391 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.16000762581825256 & Purity/1FRt*SRqqt3FBt/FRt_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 _runtime‚ 893.519104362 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.10422709584236145 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.020951736718416214 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9952903985977173 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.13953489065170288 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.43939390778541565 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.2753693759441376 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.29554590582847595 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_4‚0 (particle/accuracy‚0.7617033123970032 particle/accuracy_1_of_7‚1 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.12917324900627136 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.4545454680919647 jet/accuracy_4_of_5‚0 +particle/sensitivity‚0.4167248010635376 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.20370370149612427 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9338061809539795 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.13763228058815002 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.09389979392290115 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.1270810067653656 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.006998557597398758 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9875147342681885 ("Purity/2FRt*SRqqt*FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.0807640329003334 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.08181817829608917 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_3_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.07167655974626541 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9887213706970215 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.04891591519117355 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0003779289545491338 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.10548298060894012 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.07202756404876709 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9907243251800537 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.1680491715669632 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9952690601348877 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1585247963666916 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.07286413013935089 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.0714285746216774 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9978147149085999 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.04891591519117355 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10517241060733795 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.09841582179069519 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.0625 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.10000000149011612 jet/accuracy_6_of_7‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.1248234286904335 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9918533563613892 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.13151361048221588 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 6validation_average_jet_accuracy‚0.2743946611881256 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2682923972606659 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.10850971192121506 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.25 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.13119292259216309 particle/accuracy_3_of_4‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.17140379548072815 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09817462414503098 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08392007648944855 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.13201744854450226 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.06360699981451035 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_3‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.10544366389513016 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.20061728358268738 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.13107596337795258 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.13240858912467957 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.1899999976158142 % _timestamp‚1.7884082831047134e+09 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.1291632205247879 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.14856433868408203 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.002785799792036414 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.01024590153247118 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.08681917190551758 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26517078280448914 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.07514619827270508 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.12074830383062363 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_2_of_5‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_4_of_4‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_6‚1 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.12486596405506134 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9926981925964355 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 jet/accuracy_5_of_5‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9976012706756592 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.24923323094844818 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9691091775894165 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 9 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.061302684247493744 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.125 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9988662600517273 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.19913794100284576 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_6‚1 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09138651937246323 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9914901852607727 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08531393110752106 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.2483602911233902 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_4‚0.15426862239837646 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.12117090821266174 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09842006862163544 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.2816145420074463 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.10555555671453476 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9210527539253235 particle/accuracy_4_of_6‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.1313561350107193 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.18891827762126923 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9832661151885986 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.3499999940395355 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.0833333358168602 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.6963561773300171 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.10000000149011612 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.13344024121761322 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.0426839217543602 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.12684223055839539 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09741523861885071 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.601687490940094 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.0756378173828125 + Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.1582465022802353 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.04583093151450157 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.34621378779411316 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.040781717747449875 ;"Purity/0FRt1SRqqt1FBt/event_purity‚0.059624847024679184 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.0763888880610466 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 _step‚98 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.08530998975038528 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.09091856330633163 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.11544959992170334 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.12486596405506134 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.02045890875160694 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.06230928376317024 *jet/accuracy_1_of_1‚0.2900933027267456 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 particle/accuracy_2_of_2‚1 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt3FBt/FRt_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.0453655906021595 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.0534752756357193 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.20000000298023224 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.1979166716337204 * Purity/3FRt1SRqqt*FBt/FRt_purity‚0.125 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8157181739807129 particle/accuracy_7_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.21810057759284973 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 *jet/accuracy_1_of_3‚0.6380931735038757 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.0833333358168602 epoch‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2977489233016968 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6061235666275024 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.12524880468845367 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.13849402964115143 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.24302132427692413 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.04412408545613289 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.14275267720222473 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_3‚0.09893397986888885 'particle/f_score‚0.3530544936656952 9 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.059259261935949326 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9971330761909485 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.2027938961982727 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.0902777835726738 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 particle/accuracy_4_of_5‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0555555559694767 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.0763888880610466 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_6‚0.01923076994717121 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.1964285671710968 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.05779558792710304 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9915229678153992 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.17910632491111755 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.1458333283662796 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.12174329161643982 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.3125 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.09783609211444855 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_2_of_7‚1 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/2FRt0SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9949217438697815 ("Purity/1FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.03374934569001198 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.15255288779735565 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896174669265747 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.14954650402069092 jet/accuracy_4_of_6‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.11580733954906464 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.004692082293331623 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.17490054666996002 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.09565415233373642 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.0833333358168602 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9886640310287476 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.12934045493602753 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.15826356410980225 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08294280618429184 +jet/accuracy_2_of_2‚0.04745442420244217 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.33707699179649353 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9861221313476562 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0201774463057518 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.07519309967756271 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.07027027010917664 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.12452448159456253 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_5‚0.018010752275586128 & Purity/2FRt3SRqqt1FBt/FRt_purity‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.34732145071029663 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.0863095223903656 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.10225225239992142 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 particle/accuracy_6_of_7‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.2857142984867096 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_7‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.15017448365688324 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.08729448914527893 *jet/accuracy_1_of_2‚0.5247015357017517bƒ„NŸ‘” ‚(:† áããÔÒ‹Ë™ gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.064' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8128 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•:2°û• ‚(:ð áããÔàïóœ disk.sda.in‚0 disk.drbd1777.out‚0% disk./.usageGB‚1056.2985191345215 disk.rbd3.out‚0 disk.nvme0c0n1.in‚0.03125 disk.sda2.in‚0 disk.rbd1.out‚ 0.67578125 disk.nvme0n1.out‚ 3623.53125 disk.dm-0.out‚ 3623.53125 network.sent‚ 1.565692e+06 disk.dm-1.out‚ 37.1640625' proc.memory.availableMB‚ 259.8359375 proc.cpu.threads‚17 network.recv‚390148 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚70.88907991260844 disk.sda2.out‚0 disk.sda3.out‚ 37.1640625 disk.sda1.in‚0 disk.nvme0n1.in‚0.03125 disk.rbd2.in‚0) proc.memory.percent‚17.57640838623047 disk.sda.out‚ 37.1640625 disk.rbd2.out‚ 20.892578125$ memory_percent‚98.41408729553223 proc.memory.rssMB‚ 2879.71875 cpu‚25.404827768117038! disk.nvme0c0n1.out‚ 3623.53125 disk.dm-0.in‚0.03125 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0س ‚(:ƒ ðããÔ¯‡ìš gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.15' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.03 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nAMh— ‚(:… ðããÔý®Õ› disk.drbd1777.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2879.80078125 network.recv‚396573 disk.sda3.out‚ 37.5390625 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.rbd2.out‚ 22.482421875 disk.sda2.in‚0 disk.sda.in‚0 disk.drbd1777.in‚0* proc.memory.percent‚17.576909065246582' disk./.usagePercent‚70.889084008729 disk.rbd1.out‚ 0.67578125 disk.nvme0n1.in‚0.03125% disk.nvme0n1.out‚3952.62841796875 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda.out‚ 37.5390625 disk.sda1.out‚0 cpu‚25.399233185281464 disk.rbd2.in‚0" disk.dm-0.out‚3952.62841796875' disk.nvme0c0n1.out‚3952.62841796875 network.sent‚ 1.688639e+06 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚0.03125 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 37.5390625 disk.rbd3.in‚0 disk.dm-0.in‚0.03125$ memory_percent‚98.41554164886475( proc.memory.availableMB‚ 259.59765625% disk./.usageGB‚1056.2985801696777^ce0˜ + _wandbruntime‚923  _runtime‚923|qÇ™ Á _step‚99 ' lr-AdamW/pg1‚0.00010103140621291866 ' lr-AdamW/pg2‚0.00010103140621291866 trainer/global_step‚1249 % _timestamp‚1.7884083133150418e+09 _runtime‚ 923.727226351x×YËš Å ' lr-AdamW/pg1‚0.00010103140621291866 ' lr-AdamW/pg2‚0.00010103140621291866 trainer/global_step‚1249 % _timestamp‚1.7884083133150418e+09 _runtime‚ 923.727226351 _step‚99c2CèÞ0› + _wandbruntime‚923  _runtime‚923¹9Ô5œ ¼O KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7197019 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start3¼ª&/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1757977266935 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37387399010304 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1758142388331 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25278711 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37051105683456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206391296 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16835467 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1793530736128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚35287857828864 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16835975 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41786082 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37050918681600 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37387422950400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7196513 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39180953686528 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230331392 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1793324344832 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚35324795153515 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚35324612992759 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 trainer/global_step‚1249 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16507371 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25298747 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚33566652765184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚35287857828864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 % _timestamp‚1.7884083133158593e+09 _step‚100 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚506 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1763060852736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9638954 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚ 923.728236093 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37050918681600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16507373 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚35287875831808 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25298747 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1763060852736 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚35287875831808 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1763229851648 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41806120 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37050918681600 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37387422950400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7196513 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39180953686528 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230331392 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1793324344832 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚35324795153515 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚35324612992759 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 trainer/global_step‚1249 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16507371 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25298747 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚33566652765184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚35287857828864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 % _timestamp‚1.7884083133158593e+09 _step‚100 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚506 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1763060852736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9638954 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚ 923.728236093 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37050918681600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16507373 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚35287875831808 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25298747 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1763060852736 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚35287875831808 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1763229851648 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41806120 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37416335898624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9646331 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1764623003136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9646333 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _runtime‚ 924.37799161 _step‚101 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚33592595947520 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚35352127500603 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16520032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1764623003136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7201804 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚33592578908160 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7202311 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16520030 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25318560 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39211049226240 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41838592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236609536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37079768199680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25298498 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212669440 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37079768199680 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41838592 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16848135 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25318560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37079581184512 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41818528 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1794949937152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1794737267712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1759531553083 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41818528 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41818528 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batð!s>ch_end/inactive_split_bytes.large_pool.allocated‚37416335898624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9646331 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1764623003136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9646333 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _runtime‚ 924.37799161 _step‚101 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚33592595947520 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚35352127500603 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16520032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1764623003136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7201804 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚33592578908160 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7202311 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16520030 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25318560 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39211049226240 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41838592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236609536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37079768199680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25298498 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212669440 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37079768199680 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41838592 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16848135 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25318560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37079581184512e…çF0¡ + _wandbruntime‚924  _runtime‚924á6<S¢ Í /loss/FRt3/assignment_loss‚4.574256656304845 1loss/FBt4/assignment_loss‚0.43611031928594457 _step‚102 /loss/FRt2/assignment_loss‚4.363907486200333 2loss/SRqqt1/assignment_loss‚2.6653508795632255 -loss/FBt1/detection_loss‚6.73151757532642 %loss/total_loss‚79.01478691907525 _runtime‚ 924.379102593 /loss/FRt1/assignment_loss‚4.961731402770333 2loss/SRqqt2/assignment_loss‚2.4302960612944196 /loss/FRt3/detection_loss‚2.8498512950708283 .loss/FBt2/detection_loss‚8.519507904847464 % _timestamp‚1.7884083139670367e+09 .loss/FRt4/assignment_loss‚6.21978057058234 0loss/FBt2/assignment_loss‚0.6052759240070978 .loss/FRt2/detection_loss‚2.690031242689916 1loss/SRqqt4/detection_loss‚3.5313782592614493 1loss/FBt1/assignment_loss‚0.32771057593622355 1loss/SRqqt2/detection_loss‚2.5605357733688185 epoch‚1 /loss/FRt1/detection_loss‚3.3483803816463635 .loss/FBt3/detection_loss‚2.506070652627706 1loss/FBt3/assignment_loss‚0.40611628637587927 /loss/SRqqt1/detection_loss‚2.20232000185387 0loss/SRqqt3/detection_loss‚2.314752443775521 2loss/SRqqt3/assignment_loss‚2.7390522915741493 2loss/SRqqt4/assignment_loss‚2.9166635962632985 /loss/FRt4/detection_loss‚2.9135804427297494 .loss/FBt4/detection_loss‚6.200608895719052 trainer/global_step‚1249e´InW£ Ñ 1loss/SRqqt4/detection_loss‚3.5313782592614493 1loss/FBt1/assignment_loss‚0.32771057593622355 1loss/SRqqt2/detection_loss‚2.5605357733688185 epoch‚1 /loss/FRt1/detection_loss‚3.3483803816463635 .loss/FBt3/detection_loss‚2.506070652627706 1loss/FBt3/assignment_loss‚0.40611628637587927 /loss/SRqqt1/detection_loss‚2.20232000185387 0loss/SRqqt3/detection_loss‚2.314752443775521 2loss/SRqqt3/assignment_loss‚2.7390522915741493 2loss/SRqqt4/assignment_loss‚2.9166635962632985 /loss/FRt4/detection_loss‚2.9135804427297494 .loss/FBt4/detection_loss‚6.200608895719052 trainer/global_step‚1249 /loss/FRt3/assignment_loss‚4.574256656304845 1loss/FBt4/assignment_loss‚0.43611031928594457 _step‚102 /loss/FRt2/assignment_loss‚4.363907486200333 2loss/SRqqt1/assignment_loss‚2.6653508795632255 -loss/FBt1/detection_loss‚6.73151757532642 %loss/total_loss‚79.01478691907525 _runtime‚ 924.379102593 /loss/FRt1/assignment_loss‚4.961731402770333 2loss/SRqqt2/assignment_loss‚2.4302960612944196 /loss/FRt3/detection_loss‚2.8498512950708283 .loss/FBt2/detection_loss‚8.519507904847464 % _timestamp‚1.7884083139670367e+09 .loss/FRt4/assignment_loss‚6.21978057058234 0loss/FBt2/assignment_loss‚0.6052759240070978 .loss/FRt2/detection_loss‚2.690031242689916fì]•¢‘¤ ‚(:† ÿããÔȵ¹™ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.631' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ\Ú)¥ ‚(:‡ ÿããÔÉïêœ disk.sda1.out‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 38.33984375 disk.rbd1.in‚0% disk./.usageGB‚1056.2985801696777 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.03125 disk.sda3.out‚ 38.33984375 disk.rbd1.out‚ 0.67578125' proc.memory.availableMB‚ 258.3515625 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.dm-0.in‚0.03125$ memory_percent‚98.42314720153809 network.sent‚ 1.694124e+06 network.recv‚400281 disk.dm-1.out‚ 38.33984375 disk.rbd2.out‚ 22.607421875 disk.rbd3.in‚0 disk.drbd1777.in‚0" disk.dm-0.out‚3953.43408203125 proc.cpu.threads‚17% disk.nvme0n1.out‚3953.43408203125# proc.memory.rssMB‚ 2879.80078125 cpu‚25.393682070226824' disk./.usagePercent‚70.889084008729' disk.nvme0c0n1.out‚3953.43408203125 disk.sda3.in‚0 disk.nvme0n1.in‚0.03125* proc.memory.percent‚17.576909065246582³¶÷¦ ‚(:‰ ŽäãÔ„Ýðœ cpu‚25.388620308740435% disk./.usageGB‚1055.0738563537598 disk.rbd2.in‚0 disk.rbd2.out‚ 22.619140625* proc.memory.percent‚17.576909065246582% disk.nvme0n1.out‚4052.63330078125 disk.sda.in‚0' proc.memory.availableMB‚ 258.0703125' disk.nvme0c0n1.out‚4052.63330078125 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 38.94140625 network.sent‚ 1.802825e+06 disk.nvme0n1.in‚0.03125 disk.dm-1.out‚ 38.94140625 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 0.67578125$ memory_percent‚98.42486381530762 disk.dm-0.in‚0.03125" disk.dm-0.out‚4052.63330078125 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.drbd1777.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda.out‚ 38.94140625# proc.memory.rssMB‚ 2879.80078125 network.recv‚407489) disk./.usagePercent‚70.80689176583101 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0.03125úrM]§ ‚(:„ ŽäãÔ扗³ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚92.245' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.449 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R×-•0¨ + _wandbruntime‚958  _runtime‚958DGÈ©  ' lr-AdamW/pg1‚0.00010101772082523591 ' lr-AdamW/pg2‚0.00010101772082523591 trainer/global_step‚1299 % _timestamp‚1.7884083478025048e+09 _runtime‚ 958.214679784 _step‚103G S̪ Æ ' lr-AdamW/pg1‚0.00010101772082523591 ' lr-AdamW/pg2‚0.00010101772082523591 trainer/global_step‚1299 % _timestamp‚1.7884083478025048e+09 _runtime‚ 958.214679784 _step‚103gýʦ.0« + _wandbruntime‚958  _runtime‚958ž¦KÄ'¬ ¾O VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622848 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚34863794842624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43431772 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 % _timestamp‚1.7884083478032618e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚34863811881984 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1827614206739 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚38484058241024 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26271413 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1832901174272 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7464580 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36691426088723 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1827449085343 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1832732175360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17140323 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26291449 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36651344068608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36651326065664 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 _step‚104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚36651344068608 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚36691243927967 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 958.215652546 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43411734 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17472384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚38484058241024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17140321 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26271413 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚38484245242880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1864287431680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚38833046421504 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚38484058241024 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26271413 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1832901174272 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7464580 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36691426088723 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1827449085343 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1832732175360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17140323 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26291449 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36651344068608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36651326065664 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 _step‚104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚36651344068608 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚36691243927967 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 958.215652546 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43411734 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17472384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚38484058241024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17140321 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26271413 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚38484245242880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1864287431680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚38833046421504 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 % _timestamp‚1.7884083478032618e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚34863811881984 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1827614206739 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0hMêJ90® + _wandbruntime‚958  _runtime‚958(73jÍ&¯ ÇM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17485219 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚34889738024960 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1828838249887 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚40727665313280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43444180 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7470569 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 trainer/global_step‚1299 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10015183 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43464244 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚105 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17152982 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17152980 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _runtime‚ 958.86637802 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1865914418688 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36718576274847 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1829003371491 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26311262 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126ÙKÑBÑ&° ËM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17152980 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _runtime‚ 958.86637802 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1865914418688 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36718576274847 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1829003371491 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26311262 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17485219 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚34889738024960 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1828838249887 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚40727665313280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43444180 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7470569 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 trainer/global_step‚1299 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10015183 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43464244 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚105 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17152982 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399iˆ"[ 0± + _wandbruntime‚958  _runtime‚958¶ÈÕZP² Ê /loss/FRt2/assignment_loss‚4.785297681887944 1loss/SRqqt4/assignment_loss‚2.626285063183826 1loss/FBt1/assignment_loss‚0.23439048432497187 0loss/FBt3/assignment_loss‚0.3507193411145029 .loss/FRt3/detection_loss‚3.860462347950254 /loss/FRt1/assignment_loss‚4.587207041003487 0loss/FRt3/assignment_loss‚3.4541283811841694 0loss/FBt2/assignment_loss‚0.5094611562358645 0loss/SRqqt3/detection_loss‚2.165233754498699 /loss/SRqqt4/detection_loss‚3.04444276638653 /loss/FBt3/detection_loss‚1.6098329744794613 % _timestamp‚1.7884083484557476e+09 /loss/FRt4/assignment_loss‚6.034343810642467 2loss/SRqqt3/assignment_loss‚2.7957149018843968 .loss/FBt1/detection_loss‚4.314389312459577 1loss/SRqqt1/assignment_loss‚2.838697930177053 /loss/FBt2/detection_loss‚2.0331645611681775 _step‚106 %loss/total_loss‚66.93272019543235 trainer/global_step‚1299 2loss/SRqqt2/assignment_loss‚2.0935813334855164 0loss/SRqqt1/detection_loss‚2.046593261885697 .loss/FBt4/detection_loss‚5.595066086812452 _runtime‚ 958.867827922 .loss/FRt4/detection_loss‚2.422307740239536 epoch‚1 /loss/FRt2/detection_loss‚3.1166197887311378 /loss/FRt1/detection_loss‚3.3412760211662813 0loss/SRqqt2/detection_loss‚2.594396807930686 /loss/FBt4/assignment_loss‚0.479107646599666ȇïÇT³ Î /loss/FBt4/assignment_loss‚0.479107646599666 /loss/FRt2/assignment_loss‚4.785297681887944 1loss/SRqqt4/assignment_loss‚2.626285063183826 1loss/FBt1/assignment_loss‚0.23439048432497187 0loss/FBt3/assignment_loss‚0.3507193411145029 .loss/FRt3/detection_loss‚3.860462347950254 /loss/FRt1/assignment_loss‚4.587207041003487 0loss/FRt3/assignment_loss‚3.4541283811841694 0loss/FBt2/assignment_loss‚0.5094611562358645 0loss/SRqqt3/detection_loss‚2.165233754498699 /loss/SRqqt4/detection_loss‚3.04444276638653 /loss/FBt3/detection_loss‚1.6098329744794613 % _timestamp‚1.7884083484557476e+09 /loss/FRt4/assignment_loss‚6.034343810642467 2loss/SRqqt3/assignment_loss‚2.7957149018843968 .loss/FBt1/detection_loss‚4.314389312459577 1loss/SRqqt1/assignment_loss‚2.838697930177053 /loss/FBt2/detection_loss‚2.0331645611681775 _step‚106 %loss/total_loss‚66.93272019543235 trainer/global_step‚1299 2loss/SRqqt2/assignment_loss‚2.0935813334855164 0loss/SRqqt1/detection_loss‚2.046593261885697 .loss/FBt4/detection_loss‚5.595066086812452 _runtime‚ 958.867827922 .loss/FRt4/detection_loss‚2.422307740239536 epoch‚1 /loss/FRt2/detection_loss‚3.1166197887311378 /loss/FRt1/detection_loss‚3.3412760211662813 0loss/SRqqt2/detection_loss‚2.594396807930686j 6´!œ´ ‚(:‘ äãÔͯʙ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.862' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.972399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í1>^µ ‚(:ü äãÔ¶µ% disk./.usageGB‚1055.2366561889648 disk.rbd2.out‚22.77197265625 disk.dm-0.in‚0.03125 disk.sda3.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 0.67578125" disk.nvme0n1.out‚ 4219.46484375$ memory_percent‚98.43723773956299( proc.memory.availableMB‚ 256.04296875* proc.memory.percent‚17.576932907104492 disk.sda.out‚ 39.578125 disk.drbd1777.in‚0) disk./.usagePercent‚70.81781739936979 disk.sda3.out‚ 39.578125 disk.sda.in‚0 disk.drbd1777.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 39.578125" proc.memory.rssMB‚ 2879.8046875 disk.sda2.out‚0 disk.dm-0.out‚ 4219.46484375 network.recv‚411889 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚0.03125 disk.sda2.in‚0 disk.nvme0n1.in‚0.03125 disk.rbd3.in‚0$ disk.nvme0c0n1.out‚ 4219.46484375 cpu‚25.383179177335546 proc.cpu.threads‚17 network.sent‚ 1.808941e+06‹I%C‘¶ ‚(:† ¬äãÔÿÀ™ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.064' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8128 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ۉƷ ‚(:… ¬äãÔÒ뱜% disk./.usageGB‚1055.2371101379395 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda.out‚ 40.01171875 cpu‚25.378233042492283 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 0.67578125 network.recv‚418244 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0% proc.memory.availableMB‚ 255.59375! disk.dm-0.out‚4219.8056640625 disk.rbd2.out‚22.78759765625 disk.rbd3.out‚0 disk.sda3.in‚0 disk.nvme0n1.in‚0.03125* proc.memory.percent‚17.576932907104492 disk.nvme0c0n1.in‚0.03125 disk.sda2.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚0.03125$ memory_percent‚98.43997955322266 disk.rbd0.out‚0& disk.nvme0c0n1.out‚4219.8056640625 disk.sda3.out‚ 40.01171875 disk.sda1.out‚0" proc.memory.rssMB‚ 2879.8046875 network.sent‚ 1.916505e+06) disk./.usagePercent‚70.81784786426635 disk.rbd0.in‚0 disk.dm-1.out‚ 40.01171875$ disk.nvme0n1.out‚4219.8056640625 proc.cpu.threads‚17+s_¸ ‚(:… »äãÔ› »™ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.485' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.297 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«Ñ† ¹ ‚(:ÿ »äãÔêö¡œ disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda.in‚0* proc.memory.percent‚17.576956748962402 cpu‚25.373387091838545 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚40.65625& disk.nvme0c0n1.out‚4220.3857421875$ disk.nvme0n1.out‚4220.3857421875 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda2.in‚0$ memory_percent‚98.43907356262207 disk.sda.out‚40.65625# proc.memory.rssMB‚ 2879.80859375 disk.rbd3.out‚0 network.sent‚ 1.922281e+06 network.recv‚421792 disk.rbd1.out‚ 0.67578125! disk.dm-0.out‚4220.3857421875 disk.rbd0.out‚0' proc.memory.availableMB‚ 255.7421875 disk.sda1.in‚0 disk.nvme0n1.in‚0.03125 disk.dm-1.in‚0% disk./.usageGB‚1055.2372131347656 disk.drbd1777.in‚0 disk.rbd2.out‚22.85009765625 disk.nvme0c0n1.in‚0.03125 disk.sda3.in‚0 disk.sda3.out‚40.65625) disk./.usagePercent‚70.81785477646979 disk.dm-0.in‚0.03125ÎX\$0º + _wandbruntime‚993  _runtime‚993å±ÐÈ»  ' lr-AdamW/pg1‚0.00010099351124379169 ' lr-AdamW/pg2‚0.00010099351124379169 trainer/global_step‚1349 % _timestamp‚1.7884083825920107e+09 _runtime‚ 993.004174178 _step‚107<‘~ ̼ Æ % _timestamp‚1.7884083825920107e+09 _runtime‚ 993.004174178 _step‚107 ' lr-AdamW/pg1‚0.00010099351124379169 ' lr-AdamW/pg2‚0.00010099351124379169 trainer/global_step‚1349k´nBá0½ + _wandbruntime‚993  _runtime‚993E§eÓξ ¾O RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17773271 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27264115 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚38014794302464 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚38057874863175 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1902403497984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1935218200064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1902572496896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40278693832704 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10376654 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18113463 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682752 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _runtime‚ 993.005215991 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39917197800448 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 % _timestamp‚1.7884083825927637e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚38014812305408 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18114005 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1902403497984 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622848 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17773273 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45057424 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1897086025147 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17773273 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7737349 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚38014812305408 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27264115 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7736809 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚42213912032768 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248R(\zÈ'¿ ÂO UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚39917384802304 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40278717772800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚38057874863175 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1902403497984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1935218200064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1902572496896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40278693832704 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10376654 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18113463 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682752 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _runtime‚ 993.005215991 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39917197800448 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 % _timestamp‚1.7884083825927637e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚38014812305408 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18114005 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1902403497984 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622848 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17773273 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45057424 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1897086025147 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17773273 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7737349 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚38014812305408 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27264115 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7736809 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚42213912032768 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17773271 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27264115 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚38014794302464 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45089896 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27283902 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7742368 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1898475189899 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40307630721024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚ 993.646795683 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚38042081670144 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚38085389371019 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36186897141760 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚39946047318528 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1349 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10384033 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚39946047318528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27303964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1898310068295 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚38042081670144 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚38042063667200 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1903796636160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18126906 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚36186914181120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10384031 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884083832343152e+09 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1936843089920 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45069832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚507 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18126399 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚42244241395712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208475136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17785930 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7742873 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45069832 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17785932 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1903965648384 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1903796636160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚38085207210055 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17785930 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40307606780928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062-û4)! ÌM YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36186897141760 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚39946047318528 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1349 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10384033 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚39946047318528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27303964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1898310068295 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚38042081670144 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚38042063667200 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1903796636160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18126906 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚36186914181120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10384031 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884083832343152e+09 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1936843089920 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45069832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚507 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18126399 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 WDDeviceStatsMonitor.rJa± on_train_batch_end/inactive_split_bytes.all.freed‚42244241395712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208475136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17785930 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7742873 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45069832 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17785932 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1903965648384 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1903796636160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚38085207210055 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17785930 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40307606780928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27283902 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚39945860303360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45089896 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27283902 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7742368 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1898475189899 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40307630721024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚ 993.646795683 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚38042081670144 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚38085389371019mj[÷W0à + _wandbruntime‚993  _runtime‚993’+ àMÄ Ç 2loss/SRqqt4/assignment_loss‚2.7646600008010864 .loss/FRt2/detection_loss‚2.840650138529864 .loss/FBt1/detection_loss‚3.969289880810362 .loss/FBt4/detection_loss‚5.181396588005803 1loss/SRqqt2/assignment_loss‚2.265422505459615 0loss/FBt2/assignment_loss‚0.5438521489354158 0loss/FBt3/assignment_loss‚0.3374612506607346 0loss/SRqqt2/detection_loss‚2.493117988907865 /loss/FBt3/detection_loss‚1.3965463298646004 epoch‚1 _runtime‚ 993.648136689 1loss/SRqqt3/detection_loss‚2.4022379738635307 % _timestamp‚1.7884083832357903e+09 .loss/FRt4/detection_loss‚3.126741836147924 /loss/SRqqt4/detection_loss‚2.75222441979817 /loss/FBt2/detection_loss‚2.5535959980704566 /loss/FRt1/assignment_loss‚5.254291204696006 /loss/FRt3/assignment_loss‚3.100440190597014 1loss/SRqqt3/assignment_loss‚2.895167176685636 1loss/FBt1/assignment_loss‚0.30144042225120676 0loss/FBt4/assignment_loss‚0.6656456423410088 .loss/FRt3/detection_loss‚3.028100900352001 %loss/total_loss‚67.28814399059702 trainer/global_step‚1349 0loss/FRt2/assignment_loss‚4.6570125471461905 0loss/SRqqt1/detection_loss‚4.026108061377492 -loss/FRt1/detection_loss‚3.18278239952757 _step‚110 .loss/FRt4/assignment_loss‚5.00734329992725 1loss/SRqqt1/assignment_loss‚2.542615085840225R¼èQÅ Ë .loss/FRt4/detection_loss‚3.126741836147924 /loss/SRqqt4/detection_loss‚2.75222441979817 /loss/FBt2/detection_loss‚2.5535959980704566 /loss/FRt1/assignment_loss‚5.254291204696006 /loss/FRt3/assignment_loss‚3.100440190597014 1loss/SRqqt3/assignment_loss‚2.895167176685636 1loss/FBt1/assignment_loss‚0.30144042225120676 0loss/FBt4/assignment_loss‚0.6656456423410088 .loss/FRt3/detection_loss‚3.028100900352001 %loss/total_loss‚67.28814399059702 trainer/global_step‚1349 0loss/FRt2/assignment_loss‚4.6570125471461905 0loss/SRqqt1/detection_loss‚4.026108061377492 -loss/FRt1/detection_loss‚3.18278239952757 _step‚110 .loss/FRt4/assignment_loss‚5.00734329992725 1loss/SRqqt1/assignment_loss‚2.542615085840225 2loss/SRqqt4/assignment_loss‚2.7646600008010864 .loss/FRt2/detection_loss‚2.840650138529864 .loss/FBt1/detection_loss‚3.969289880810362 .loss/FBt4/detection_loss‚5.181396588005803 1loss/SRqqt2/assignment_loss‚2.265422505459615 0loss/FBt2/assignment_loss‚0.5438521489354158 0loss/FBt3/assignment_loss‚0.3374612506607346 0loss/SRqqt2/detection_loss‚2.493117988907865 /loss/FBt3/detection_loss‚1.3965463298646004 epoch‚1 _runtime‚ 993.648136689 1loss/SRqqt3/detection_loss‚2.4022379738635307 % _timestamp‚1.7884083832357903e+09n ißà‘Æ ‚(:† ÊäãÔÔºš gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.214' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2428 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X͇¶Ç ‚(:ˆ ÊäãÔªˆ–›% disk./.usageGB‚1055.2372131347656 disk.dm-1.in‚0$ memory_percent‚98.43974113464355 cpu‚25.368732237307174 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd2.in‚0& disk.nvme0c0n1.out‚4220.7373046875 disk.sda2.out‚0* proc.memory.percent‚17.576956748962402 network.recv‚426248 disk.nvme0n1.in‚0.03125 disk.dm-1.out‚ 41.08203125 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 network.sent‚ 1.928323e+06 disk.sda.in‚0 disk.rbd2.out‚24.44775390625 disk.sda.out‚ 41.08203125 disk.rbd1.in‚0 disk.rbd3.out‚0! disk.dm-0.out‚4220.7373046875 disk.nvme0c0n1.in‚0.03125 disk.sda3.in‚0 disk.sda3.out‚ 41.08203125' proc.memory.availableMB‚ 255.6328125# proc.memory.rssMB‚ 2879.80859375) disk./.usagePercent‚70.81785477646979 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 0.67578125$ disk.nvme0n1.out‚4220.7373046875 disk.dm-0.in‚0.03125 +ð%‘È ‚(:† ÙäãÔÒέ™ gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.926' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7852 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ïh³âýÉ ‚(:ò ÙäãÔŒ•Ù› disk.dm-0.in‚0.03125 disk.rbd3.out‚0 disk.sda2.out‚0 disk.drbd1777.in‚0# proc.memory.rssMB‚ 2879.80859375 cpu‚25.36412878422046 network.sent‚ 2.036695e+06 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0" disk./.usageGB‚1055.2373046875 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.576956748962402 proc.cpu.threads‚17& disk.nvme0c0n1.out‚4221.4189453125 network.recv‚433206$ disk.nvme0n1.out‚4221.4189453125 disk.dm-1.out‚41.75 disk.rbd2.in‚0 disk.sda3.out‚41.75 disk.nvme0n1.in‚0.03125 disk.nvme0c0n1.in‚0.03125 disk.sda.out‚41.75 disk.drbd1777.out‚0# memory_percent‚98.4412431716919( proc.memory.availableMB‚ 255.38671875) disk./.usagePercent‚70.81786092065062! disk.dm-0.out‚4221.4189453125 disk.rbd0.out‚0 disk.rbd2.out‚24.58837890625 disk.sda1.out‚0 disk.rbd1.out‚ 0.67578125©Ï>^2Ê - _wandbruntime‚1027  _runtime‚1027J}´ÈË Â % _timestamp‚1.7884084169585958e+09 _runtime‚ 1027.37076137 _step‚111 ' lr-AdamW/pg1‚0.00010095878251382647 ' lr-AdamW/pg2‚0.00010095878251382647 trainer/global_step‚1399^éñàÌÌ Æ ' lr-AdamW/pg2‚0.00010095878251382647 trainer/global_step‚1399 % _timestamp‚1.7884084169585958e+09 _runtime‚ 1027.37076137 _step‚111 ' lr-AdamW/pg1‚0.00010095878251382647oRTD2Í - _wandbruntime‚1027  _runtime‚1027Ç&xMÅ'Î ¿O L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28256817 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41724365184000 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 % _timestamp‚1.7884084169593813e+09 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚39424505798383 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18406221 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18406221 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28256817 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚578 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18754256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _runtime‚1027.371701331 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18406223 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1966392722159 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚576 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622848 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚41350524361728 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚39378262539264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41350337359872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8008752 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚37458130115584 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚1399 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46663038 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28276853 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28276853 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46683076 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚37458113076224 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2006363796992 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18406223 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1966557843555 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41350524361728 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚39378262539264 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8009328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2006151114240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1972243819520 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚39378280542208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10745506 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1972074820608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41350337359872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41724341243904ݯtJÉ'Ï ÃO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41350337359872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41724341243904 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28256817 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41724365184000 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 % _timestamp‚1.7884084169593813e+09 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚39424505798383 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18406221 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18406221 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28256817 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚578 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18754256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _runtime‚1027.371701331 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18406223 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1966392722159 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚576 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622848 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚41350524361728 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚39378262539264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41350337359872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8008752 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚37458130115584 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚1399 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46663038 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28276853 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28276853 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46683076 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚37458113076224 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2006363796992 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18406223 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1966557843555 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41350524361728 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚39378262539264 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8009328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2006151114240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1972243819520 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚39378280542208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10745506 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1972074820608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038pK‡N2Ð - _wandbruntime‚1028  _runtime‚1028á¾ *' Ñ ÉM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208475136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39405549906944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41753254192128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1967781886703 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚1399 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28276604 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 _step‚113 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28296666 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚39405549906944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1973467958784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18767049 QEDeviceil½Ç¨StatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8014168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18418882 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39405531904000 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1973467958784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚575 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚37484056258560 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28296666 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1967947008307 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232415232 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43760826788864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39451838145263 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1973636971008 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1973636971008 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46695484 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884084176140678e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18418880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18418882 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8014743 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41753278132224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚39452020306227 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚41378999862784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10752883 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46715548 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46695484 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚41379186877952 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28276604 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46715548 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚577 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18418880 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2007781071872†ö^CÓ&Ò ÍM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43760826788864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39451838145263 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1973636971008 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1973636971008 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46695484 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884084176140678e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18418880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18418882 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8014743 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41753278132224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚39452020306227 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚41378999862784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10752883 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46715548 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46695484 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚41379186877952 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28276604 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46715548 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚577 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18418880 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2007781071872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208475136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39405549906944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41753254192128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1967781886703 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚1399 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28276604 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 _step‚113 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28296666 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚39405549906944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1973467958784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18767049 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8014168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18418882 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39405531904000 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1973467958784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚575 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚37484056258560 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28296666 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1967947008307 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232415232 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0qi<2Ó - _wandbruntime‚1028  _runtime‚1028\èSÔ Í 2loss/SRqqt4/assignment_loss‚2.5950165625774497 /loss/SRqqt4/detection_loss‚3.83571846557386 trainer/global_step‚1399 .loss/FRt2/detection_loss‚2.839778552452723 1loss/SRqqt2/assignment_loss‚2.068765173355738 0loss/FBt2/assignment_loss‚0.5940984344730774 0loss/FBt3/assignment_loss‚0.3453170879400507 /loss/FRt1/detection_loss‚3.2682866553465524 /loss/FBt1/detection_loss‚3.8568028405760275 .loss/FRt4/detection_loss‚3.178247799475988 .loss/FBt2/detection_loss‚2.648903702839258 % _timestamp‚1.7884084176157045e+09 /loss/FRt4/assignment_loss‚5.813067575295766 1loss/SRqqt1/detection_loss‚3.1105210565030577 1loss/SRqqt3/detection_loss‚2.3416271809767752 /loss/FRt2/assignment_loss‚4.051554733631658 /loss/FRt3/detection_loss‚2.7955464205394187 1loss/SRqqt2/detection_loss‚2.4320051334798336 /loss/FRt3/assignment_loss‚4.153102792799473 0loss/FBt1/assignment_loss‚0.3014113686972864 0loss/FBt4/assignment_loss‚0.5741527058649808 .loss/FBt4/detection_loss‚5.715738495439291 %loss/total_loss‚67.79524564615417 epoch‚1 _runtime‚1028.027728882 2loss/SRqqt3/assignment_loss‚2.4858866951295306 .loss/FBt3/detection_loss‚1.482505478754528 _step‚114 /loss/FRt1/assignment_loss‚4.981051677749271 2loss/SRqqt1/assignment_loss‚2.3261390566825866øgWÕ Ñ 2loss/SRqqt4/assignment_loss‚2.5950165625774497 /loss/SRqqt4/detection_loss‚3.83571846557386 trainer/global_step‚1399 .loss/FRt2/detection_loss‚2.839778552452723 1loss/SRqqt2/assignment_loss‚2.068765173355738 0loss/FBt2/assignment_loss‚0.5940984344730774 0loss/FBt3/assignment_loss‚0.3453170879400507 /loss/FRt1/detection_loss‚3.2682866553465524 /loss/FBt1/detection_loss‚3.8568028405760275 .loss/FRt4/detection_loss‚3.178247799475988 .loss/FBt2/detection_loss‚2.648903702839258 % _timestamp‚1.7884084176157045e+09 /loss/FRt4/assignment_loss‚5.813067575295766 1loss/SRqqt1/detection_loss‚3.1105210565030577 1loss/SRqqt3/detection_loss‚2.3416271809767752 /loss/FRt2/assignment_loss‚4.051554733631658 /loss/FRt3/detection_loss‚2.7955464205394187 1loss/SRqqt2/detection_loss‚2.4320051334798336 /loss/FRt3/assignment_loss‚4.153102792799473 0loss/FBt1/assignment_loss‚0.3014113686972864 0loss/FBt4/assignment_loss‚0.5741527058649808 .loss/FBt4/detection_loss‚5.715738495439291 %loss/total_loss‚67.79524564615417 epoch‚1 _runtime‚1028.027728882 2loss/SRqqt3/assignment_loss‚2.4858866951295306 .loss/FBt3/detection_loss‚1.482505478754528 _step‚114 /loss/FRt1/assignment_loss‚4.981051677749271 2loss/SRqqt1/assignment_loss‚2.3261390566825866rdQåúÖ ‚(:… èäãÔ’†Á™ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚94.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8842 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤¿ × ‚(:ÿ èäãÔûÐà› disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 42.2421875 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 42.2421875# disk.nvme0n1.out‚4221.857421875$ memory_percent‚98.44000339508057 network.recv‚437282 disk.dm-1.in‚0 disk.dm-1.out‚ 42.2421875 disk.rbd0.in‚0 disk.rbd2.out‚24.58837890625 disk.sda3.in‚0 disk.dm-0.in‚0.03125 disk.drbd1777.out‚0 disk.nvme0n1.in‚0.03125 network.sent‚ 2.042336e+06" disk./.usageGB‚1055.2373046875% disk.nvme0c0n1.out‚4221.857421875 disk.sda.in‚0( proc.memory.availableMB‚ 255.58984375* proc.memory.percent‚17.576956748962402 disk.nvme0c0n1.in‚0.03125 disk.rbd1.out‚ 0.67578125# proc.memory.rssMB‚ 2879.80859375 cpu‚25.35967907930785 proc.cpu.threads‚17) disk./.usagePercent‚70.81786092065062 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda1.in‚0 disk.dm-0.out‚4221.857421875y ‘Ø ‚(:† ÷äãÔ³¨ó™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.889' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E!W¼Ù ‚(:‡ ÷äãÔÕ©Û›) disk./.usagePercent‚70.81785964061294 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚0.03125 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0 cpu‚25.355125746692785 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 42.80859375 disk.sda1.out‚0 disk.dm-0.in‚0.03125 disk.rbd2.in‚0 disk.rbd2.out‚ 24.7607421875' disk.nvme0c0n1.out‚4222.24560546875 disk.dm-1.in‚0# memory_percent‚98.4415054321289% proc.memory.availableMB‚ 255.34375* proc.memory.percent‚17.576956748962402% disk./.usageGB‚1055.2372856140137 disk.nvme0c0n1.in‚0.03125 disk.sda3.in‚0 disk.sda3.out‚ 42.80859375 disk.drbd1777.out‚0 disk.dm-1.out‚ 42.80859375# proc.memory.rssMB‚ 2879.80859375 proc.cpu.threads‚17 network.sent‚ 2.149949e+06 network.recv‚443098% disk.nvme0n1.out‚4222.24560546875 disk.rbd3.in‚0 disk.rbd1.out‚ 0.67578125" disk.dm-0.out‚4222.24560546875‹êTÝ2Ú - _wandbruntime‚1061  _runtime‚10612øYÉÛ Ã _step‚115 ' lr-AdamW/pg1‚0.00010091354187275554 ' lr-AdamW/pg2‚0.00010091354187275554 trainer/global_step‚1449 % _timestamp‚1.7884084513960288e+09 _runtime‚1061.808192902¢AæÍÜ Ç _step‚115 ' lr-AdamW/pg1‚0.00010091354187275554 ' lr-AdamW/pg2‚0.00010091354187275554 trainer/global_step‚1449 % _timestamp‚1.7884084513960288e+09 _runtime‚1061.808192902sp+>Ç2Ý - _wandbruntime‚1061  _runtime‚1061íhJ¸$Þ ¿O L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2041915142144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19039173 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚40791136733591 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚40741748779008 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚40741730776064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2077322974208 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11114354 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29269555 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚40741730776064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚42783663921152 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45247335569408 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚42783663921152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29249519 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43170012595200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚40741748779008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚553 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8281573 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2036029661963 _runtime‚1061.809185511 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11114356 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48308728 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚38755289232384 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚1449 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42783476919296 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚551 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2041915142144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230331392 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2035864540567 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19039171 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29269555 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2077116582912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚38755272193024 JBDeviceStatsMonitor.on_train_batch_start/segment.small_poui5jµol.current‚199 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8281022 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚45247105238016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2041746143232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2041746143232 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856Ô ª{É'ß ÃO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230331392 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2035864540567 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19039171 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29269555 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2077116582912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚38755272193024 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8281022 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚45247105238016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2041746143232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2041746143232 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2041915142144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19039173 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚40791136733591 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚40741748779008 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚40741730776064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2077322974208 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11114354 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29269555 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚40741730776064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚42783663921152 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45247335569408 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚42783663921152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29249519 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43170012595200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚40741748779008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚553 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8281573 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2036029661963 _runtime‚1061.809185511 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11114356 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48308728 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚38755289232384 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚1449 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42783476919296 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚551 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2041915142144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217tD¡cý2à - _wandbruntime‚1062  _runtime‚1062Öã9ŒÎ&á ÈM O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 _runtime‚1062.447570947 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2043139281408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚1449 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2043139281408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48321136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19051830 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2078746451456 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚40818469080471 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚40769000140800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚40769018143744 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29289368 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45277437482496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚40769000140800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2037418826715 $ _timestamp‚1.788408452034769e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43198901603328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚38781232414720 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚48321136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19408692 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8286959 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234512384 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2043308293632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19408173 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚38781215375360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19051832 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8286442 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11121733 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42812139422208 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29289368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210572288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11121731 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42812326437376 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚40769018143744 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43198925543424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2037253705111 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2078535879168 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45277671994880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19051832 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29269306 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48341200 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40818651241435 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42812326437376 _step‚117 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚082 ÓÒ&â ÌM R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚40818469080471 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚40769000140800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚40769018143744 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29289368 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45277437482496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚40769000140800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2037418826715 $ _timestamp‚1.788408452034769e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43198901603328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚38781232414720 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚48321136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19408692 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8286959 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234512384 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2043308293632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19408173 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚38781215375360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19051832 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8286442 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11121733 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42812139422208 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29289368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210572288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11121731 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42812326437376 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚40769018143744 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43198925543424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2037253705111 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2078535879168 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45277671994880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19051832 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29269306 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48341200 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40818651241435 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42812326437376 _step‚117 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 _runtime‚1062.447570947 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2043139281408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚1449 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2043139281408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48321136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19051830 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2078746451456u0Öú·2ã - _wandbruntime‚1062  _runtime‚1062 uRQä Ë /loss/FRt2/assignment_loss‚4.169089255854487 2loss/SRqqt3/assignment_loss‚2.4531844525921103 1loss/FBt3/assignment_loss‚0.21359216291994187 /loss/FRt2/detection_loss‚2.3020941861905158 .loss/FRt3/detection_loss‚3.347464949676865 epoch‚1 2loss/SRqqt1/assignment_loss‚2.4211114040192436 0loss/FBt1/assignment_loss‚0.6219219273171567 1loss/SRqqt2/detection_loss‚1.9699196826610992 0loss/SRqqt4/detection_loss‚2.623834753409028 #loss/total_loss‚67.587818782565 _step‚118 0loss/SRqqt1/detection_loss‚3.466437871841823 .loss/FBt2/detection_loss‚4.275964532579694 /loss/FBt3/detection_loss‚1.4816689178703564 _runtime‚1062.448982746 /loss/FRt1/assignment_loss‚4.773392083909776 0loss/FBt2/assignment_loss‚0.9643353950710402 0loss/FBt4/assignment_loss‚0.5914175778667222 .loss/FRt4/detection_loss‚2.698366787698534 /loss/FRt1/detection_loss‚3.1296163585450913 0loss/SRqqt3/detection_loss‚2.157563844079888 trainer/global_step‚1449 0loss/FRt3/assignment_loss‚3.3416979893257746 /loss/FRt4/assignment_loss‚5.562842616328487 $ _timestamp‚1.788408452036636e+09 2loss/SRqqt2/assignment_loss‚1.7120783253392178 2loss/SRqqt4/assignment_loss‚2.5500431172549725 -loss/FBt1/detection_loss‚4.91108152822212 .loss/FBt4/detection_loss‚5.849099061991039(Ï¡iõå Ï 2loss/SRqqt1/assignment_loss‚2.4211114040192436 0loss/FBt1/assignment_loss‚0.6219219273171567 1loss/SRqqt2/detection_loss‚1.9699196826610992 0loss/SRqqt4/detection_loss‚2.623834753409028 #loss/total_loss‚67.58781878256mž¸`5 _step‚118 0loss/SRqqt1/detection_loss‚3.466437871841823 .loss/FBt2/detection_loss‚4.275964532579694 /loss/FBt3/detection_loss‚1.4816689178703564 _runtime‚1062.448982746 /loss/FRt1/assignment_loss‚4.773392083909776 0loss/FBt2/assignment_loss‚0.9643353950710402 0loss/FBt4/assignment_loss‚0.5914175778667222 .loss/FRt4/detection_loss‚2.698366787698534 /loss/FRt1/detection_loss‚3.1296163585450913 0loss/SRqqt3/detection_loss‚2.157563844079888 trainer/global_step‚1449 0loss/FRt3/assignment_loss‚3.3416979893257746 /loss/FRt4/assignment_loss‚5.562842616328487 $ _timestamp‚1.788408452036636e+09 2loss/SRqqt2/assignment_loss‚1.7120783253392178 2loss/SRqqt4/assignment_loss‚2.5500431172549725 -loss/FBt1/detection_loss‚4.91108152822212 .loss/FBt4/detection_loss‚5.849099061991039 /loss/FRt2/assignment_loss‚4.169089255854487 2loss/SRqqt3/assignment_loss‚2.4531844525921103 1loss/FBt3/assignment_loss‚0.21359216291994187 /loss/FRt2/detection_loss‚2.3020941861905158 .loss/FRt3/detection_loss‚3.347464949676865 epoch‚1v—ÍàPœæ ‚(:‘ †åãÔÞâö™ gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.567' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.513399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FYó¿ç ‚(:… †åãÔ€ÎÚœ disk.sda2.out‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚4222.65966796875 disk.rbd2.in‚0$ memory_percent‚98.44162464141846# proc.memory.rssMB‚ 2879.80859375* proc.memory.percent‚17.576956748962402% disk./.usageGB‚1055.2373237609863 disk.sda3.out‚ 43.2890625 disk.rbd0.in‚0( proc.memory.availableMB‚ 255.32421875 network.recv‚448191' disk.nvme0c0n1.out‚4222.65966796875 disk.sda.out‚ 43.2890625 disk.nvme0n1.in‚0.03125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚0.03125 disk.sda1.in‚0 disk.dm-0.in‚0.03125 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 0.67578125" disk.dm-0.out‚4222.65966796875) disk./.usagePercent‚70.81786220068828 disk.drbd1777.in‚0 disk.dm-1.out‚ 43.2890625 disk.rbd2.out‚ 24.7607421875 cpu‚25.35087781067091 proc.cpu.threads‚17 network.sent‚ 2.1562e+06¨©4—œè ‚(:‘ •åãÔËÑÇ™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.171' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.834200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Íᘷé ‚(:„ •åãÔ˜þ¬œ network.recv‚454798 disk.dm-1.in‚0 disk.dm-1.out‚ 43.85546875 disk.rbd2.out‚ 26.4716796875$ memory_percent‚99.94819164276123* proc.memory.percent‚17.576980590820312) disk./.usagePercent‚70.80669771212006% disk./.usageGB‚1055.0709648132324" disk.dm-0.out‚4226.05517578125 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.67578125% disk.nvme0n1.out‚4226.05517578125 disk.nvme0c0n1.in‚0.03125 disk.sda3.out‚ 43.85546875 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.drbd1777.out‚0& proc.memory.availableMB‚ 8.48828125 cpu‚25.34658228099875 network.sent‚ 2.264489e+06 disk.nvme0n1.in‚0.03125 disk.dm-0.in‚0.03125' disk.nvme0c0n1.out‚4226.05517578125 disk.sda1.in‚0 proc.memory.rssMB‚ 2879.8125 disk.sda3.in‚0 disk.sda.out‚ 43.85546875 disk.sda1.out‚0 disk.rbd3.out‚0 disk.drbd1777.in‚0¢œCœê ‚(:‘ ¤åãÔîÕ¥™ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.996' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.999200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚R~;ë ‚(:„ ¤åãÔÌ•Í› network.recv‚458613" disk.dm-0.out‚4231.77099609375 disk.dm-1.out‚ 44.15234375' disk.nvme0c0n1.out‚4231.77099609375 proc.memory.rssMB‚ 2879.8125* proc.memory.percent‚17.576980590820312 disk.sda3.in‚0 disk.sda3.out‚ 44.15234375& proc.memory.availableMB‚ 8.32421875 cpu‚25.34264411311048 disk.dm-0.in‚0.03125 disk.rbd3.out‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚4231.77099609375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0 network.sent‚ 2.269797e+06) disk./.usagePercent‚70.80667595147965 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 0.67578125 disk.nvme0n1.in‚0.03125 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd2.out‚ 26.4716796875 disk.nvme0c0n1.in‚0.03125 disk.sda.in‚0 disk.sda.out‚ 44.15234375% disk./.usageGB‚1055.0706405639648 disk.sda2.in‚0$ memory_percent‚99.94919300079346tÆzŽ2ì - _wandbruntime‚1096  _runtime‚1096•|LÉí à ' lr-AdamW/pg1‚0.00010085779874866068 ' lr-AdamW/pg2‚0.00010085779874866068 trainer/global_step‚1499 % _timestamp‚1.7884084858004158e+09 _runtime‚1096.212587808 _step‚119s¶®Íî Ç trainer/global_step‚1499 % _timestamp‚1.7884084858004158e+09 _runtime‚1096.212587808 _step‚119 ' lr-AdamW/pg1‚0.00010085779874866068 ' lr-AdamW/pg2‚0.00010085779874866068w±ãÄ2ï - _wandbruntime‚1096  _runtime‚1096üòÇ©Å'ð ¿O \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42105217015808 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8552373 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚44615660006400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19672123 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19672121 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42105199012864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30262257 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30262257 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2148024328704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42105199012864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚44615636066304 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2148241205760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚40052448349184 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49934380 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚567 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚44216803480576 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚44216803480576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 % _timestamp‚1.7884084858012223e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19672121 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30242221 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46763660395008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2105501480371 _step‚120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11483204 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚1499 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49914342 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2111417465856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19672123 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20035579 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 _runtime‚1096.213526797 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2111586464768 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚46763901212160 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚42157767668799 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44216616478720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2111586464768 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20035012 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2105336358975 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30242221 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚44216616478720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8551808 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2111417465856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11483206 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240817152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992Éa°÷É'ñ ÃO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚44615636066304 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2148241205760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚40052448349184 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49934380 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚567 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚44216803480576 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚44216803480576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 % _timestamp‚1.7884084858012223e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19672121 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30242221 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46763660395008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2105501480371 _step‚120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11483204 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚1499 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49914342 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2111417465856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19672123 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20035579 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 _runtime‚1096.213526797 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2111586464768 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚46763901212160 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚42157767668799 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44216616478720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2111586464768 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20035012 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2105336358975 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30242221 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚44216616478720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8551808 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2111417465856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11483206 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240817152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42105217015808 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8552373 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚44615660006400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19672123 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19672121 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42105199012864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30262257 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30262257 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2148024328704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42105199012864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944xÿ™"2ò - _wandbruntime‚1096  _runtime‚10962bç¡{ó ÉM ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚40078391531520 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚40078374492160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚44644549014528 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚42185282176643 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚44245465996800 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20048280 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 _runtime‚1096.866252481 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8557697 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2149442526208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8557163 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46793991540736 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19684780 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30262008 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42132468377600 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2106890645123 trainer/global_step‚1499 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49966852 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚44245465996800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2112979616256 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2149657292800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49946788 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42132468377600 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49966852 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49946788 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _step‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30282070 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20047744 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238706688 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2112810604032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚44245278981632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2112979616256 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214766592 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0*ÁüÓ&ô ÍM ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚40078391531520 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚40078374492160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚44644549014528 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚42185282176643 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚44245465996800 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20048280 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 _runtime‚1096.866252481 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8557697 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2149442526208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8557163 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46793991540736 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19684780 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30262008 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42132468377600 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2106890645123 trainer/global_step‚1499 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49966852 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚44245465996800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2112979616256 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2149657292800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49946788 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42132468377600 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49966852 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49946788 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _step‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30282070 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20047744 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238706688 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2112810604032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚44245278981632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2112979616256 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214766592 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0yXz82õ - _wandbruntime‚1096  _runtime‚1096K"µpSö Í 1loss/SRqqt4/assignment_loss‚2.282642794066462 /loss/FRt2/detection_loss‚2.4876501388468983 1loss/SRqqt2/detection_loss‚1.5716306098465678 /loss/FRt2/assignment_loss‚3.907385575569282 1loss/FBt1/assignment_loss‚0.30163508674848816 /loss/FRt4/detection_loss‚2.4662129426824637 /loss/FBt2/detection_loss‚3.7623811066150665 /loss/FRt4/assignment_loss‚5.565778025265398 /loss/FBt2/assignment_loss‚0.809094563126564 /loss/FBt3/detection_loss‚1.3994390077315129 % _timestamp‚1.7884084864551911e+09 _runtime‚1096.867229451 1loss/SRqqt3/assignment_loss‚2.662566203697055 .loss/FBt4/detection_loss‚5.919620133936405 _step‚122 /loss/FRt3/assignment_loss‚4.174399893334571 2loss/SRqqt1/assignment_loss‚2.4002949683754533 1loss/SRqqt2/assignment_loss‚2.048063980250419 .loss/FRt3/detection_loss‚2.671596553731472 1loss/SRqqt4/detection_loss‚3.9859303112687736 trainer/global_step‚1499 0loss/FBt3/assignment_loss‚0.3898075781040394 0loss/FBt4/assignment_loss‚0.5038799484803652 .loss/FRt1/detection_loss‚3.166065656168516 1loss/SRqqt3/detection_loss‚2.2411841375028922 0loss/SRqqt1/detection_loss‚2.293100087730973 epoch‚1 /loss/FRt1/assignment_loss‚4.217935254407483 .loss/FBt1/detection_loss‚4.784333718043786 %loss/total_loss‚66.01262827553091ÑŒ W÷ Ñ /loss/FRt1/assignment_loss‚4.217935254407483 .loss/FBt1/detection_loss‚4.784333718043786 %loss/total_loss‚66.01262827553091 1loss/SRqqt4/assignment_loss‚2.282642794066462 /loss/FRt2/detection_loss‚2.4876501388468983 1loss/SRqqt2/detection_loss‚1.5716306098465678 /loss/FRt2/assignment_loss‚3.907385575569282 1loss/FBt1/assignment_loss‚0.30163508674848816 /loss/FRt4/detection_loss‚2.4662129426824637 /loss/FBt2/detection_loss‚3.7623811066150665 /loss/FRt4/assignment_loss‚5.565778025265398 /loss/FBt2/assignment_loss‚0.809094563126564 /loss/FBt3/detection_loss‚1.3994390077315129 % _timestamp‚1.7884084864551911e+09 _runtime‚1096.867229451 1loss/SRqqt3/assignment_loss‚2.662566203697055 .loss/FBt4/detection_loss‚5.919620133936405 _step‚122 /loss/FRt3/assignment_loss‚4.174399893334571 2loss/SRqqt1/assignment_loss‚2.4002949683754533 1loss/SRqqt2/assignment_loss‚2.048063980250419 .loss/FRt3/detection_loss‚2.671596553731472 1loss/SRqqt4/detection_loss‚3.9859303112687736 trainer/global_step‚1499 0loss/FBt3/assignment_loss‚0.3898075781040394 0loss/FBt4/assignment_loss‚0.5038799484803652 .loss/FRt1/detection_loss‚3.166065656168516 1loss/SRqqt3/detection_loss‚2.2411841375028922 0loss/SRqqt1/detection_loss‚2.293100087730973 epoch‚1z~¦ä‘ø ‚(:† ³åãÔ›ÁÄ™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.174' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4348 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æúeÆù ‚(:ù ³åãÔšÞ„œ% disk./.usageGB‚1055.2026290893555 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.sda1.out‚0 network.recv‚462169 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.sda3.out‚44.6875 disk.dm-0.in‚0.03125 proc.memory.rssMB‚ 2879.8125 disk.drbd1777.in‚0* proc.memory.percent‚17.576980590820312) disk./.usagePercent‚70.81553381216436 disk.dm-1.out‚44.6875 disk.rbd1.in‚0% disk.nvme0n1.out‚4365.24755859375 disk.dm-1.in‚0$ memory_percent‚99.95076656341553& proc.memory.availableMB‚ 8.06640625 proc.cpu.threads‚17 disk.rbd2.out‚ 26.5498046875 disk.nvme0c0n1.in‚0.03125' disk.nvme0c0n1.out‚4365.24755859375 network.sent‚ 2.275352e+06 disk.rbd0.out‚0 disk.sda2.out‚0 cpu‚25.338755035346406 disk.sda.out‚44.6875 disk.rbd1.out‚ 0.67578125 disk.nvme0n1.in‚0.03125" disk.dm-0.out‚4365.24755859375›]fZ‘ú ‚(:† ÂåãÔɹ™ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.102' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6204 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G«Qý û ‚(:‚ ÂåãÔ¾…ü› disk.drbd1777.in‚0 disk.sda2.in‚0 network.sent‚ 2.383818e+06 disk.sda3.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚0.03125 disk.dm-0.in‚0.03125 proc.cpu.threads‚17) disk./.usagePercent‚70.82426213303675 disk.dm-1.in‚0 disk.rbd3.in‚0$ memory_percent‚99.95081424713135& proc.memory.availableMB‚ 8.05859375 cpu‚25.334773284702866 disk.nvme0c0n1.in‚0.03125 network.recv‚469094 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.67578125! disk.dm-0.out‚4498.9091796875 disk.rbd0.in‚0 disk.rbd2.in‚0% disk./.usageGB‚1055.3326873779297 disk.sda3.out‚ 45.70703125 disk.drbd1777.out‚0$ disk.nvme0n1.out‚4498.9091796875 disk.rbd0.out‚0 disk.rbd3.out‚0* proc.memory.percent‚17.576980590820312 disk.sda.out‚ 45.70703125 disk.rbd2.out‚ 26.5498046875 proc.memory.rssMB‚ 2879.8125& disk.nvme0c0n1.out‚4498.9091796875 disk.sda.in‚0 disk.dm-1.out‚ 45.70703125 disk.sda2.out‚0OÇÖD2ü - _wandbruntime‚1131  _runtime‚1131&2ôÉý à _runtime‚1131.232637005 _step‚123 ' lr-AdamW/pg1‚0.00010079156475832542 ' lr-AdamW/pg2‚0.00010079156475832542 trainer/global_step‚1549 % _timestamp‚1.7884085208205366e+09saßÍþ Ç % _timestamp‚1.7884085208205366e+09 _runtime‚1131.232637005 _step‚123 ' lr-AdamW/pg1‚0.00010079156475832542 ' lr-AdamW/pg2‚0.00010079156475832542 trainer/global_step‚1549{;°O2ÿ - _wandbruntime‚1131  _runtime‚1131ƒÊ.Å'€¿O UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45649943040000 trainer/global_step‚1549 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚43524580764763 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚43524398604007 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51539994 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31254959 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8825065 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚124 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45649756038144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8824529 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚45649943040000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20305071 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2181257787392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884085208212068e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242914304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2181088788480 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31234923 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46061307417600 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2174973298779 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51560032 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48280284334080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚1131.233548612 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20305073 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20305071 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20305073 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31254959 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2219219830784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2219000856576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31234923 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2181257787392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚43468685252608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41349607465984 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚48280527248384 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2174808177383 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚43468685252608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2181088788480³`J¢ ÃO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41349590426624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11852054 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20676583 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚43468667249664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218974208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11852056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚43468667249664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48280284334080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚1131.233548612 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20305073 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚203¾û'05071 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20305073 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31254959 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2219219830784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2219000856576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31234923 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2181257787392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚43468685252608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41349607465984 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚48280527248384 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2174808177383 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚43468685252608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2181088788480 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45649943040000 trainer/global_step‚1549 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚43524580764763 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚43524398604007 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51539994 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31254959 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8825065 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚124 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45649756038144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8824529 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚45649943040000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20305071 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2181257787392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884085208212068e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242914304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2181088788480 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31234923 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46061307417600 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2174973298779 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51560032 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8830404 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1549 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51572440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚43495936614400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2182650938880 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51572440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11859433 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41375550648320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚1131.902529074 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2182481926656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20317730 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2176197341927 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚43495954617344 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51592504 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236609536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2176362463531 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚43495954617344 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2182650938880 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20317732 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20689314 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20317730 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚43495936614400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚46090220365824 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2220628955648 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 _step‚125 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚43551730950887 % _timestamp‚1.7884085214897287e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31254710 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11859431 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2220416286208 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212669440 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚45678605556224 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48310849321472 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41375533608960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0±›9Ó&„ÍM [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48310849321472 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41375533608960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31274772 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚48310612711936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚46090196425728 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8830404 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1549 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51572440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚43495936614400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2182650938880 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51572440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11859433 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41375550648320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚1131.902529074 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2182481926656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20317730 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2176197341927 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚43495954617344 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51592504 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236609536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2176362463531 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚43495954617344 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2182650938880 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20317732 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20689314 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20317730 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚43495936614400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚46090220365824 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2220628955648 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 _step‚125 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚43551730950887 % _timestamp‚1.7884085214897287e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31254710 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11859431 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2220416286208 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212669440 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚45678605556224}שfQ2…- _wandbruntime‚1131  _runtime‚1131ò^…–X†Ò /loss/FBt1/detection_loss‚3.8353112898766994 /loss/FBt3/detection_loss‚1.2447936750059005 _runtime‚1131.903629837 /loss/FRt4/assignment_loss‚5.383458441701428 1loss/FBt3/assignment_loss‚0.27194852079146375 /loss/FRt3/detection_loss‚2.7587156850238177 1loss/SRqqt3/detection_loss‚1.8253111589346558 -loss/FBt4/detection_loss‚4.67413841382317 %loss/total_loss‚61.24410905555898 epoch‚1 trainer/global_step‚1549 1loss/SRqqt4/assignment_loss‚2.344609876473745 0loss/FBt2/assignment_loss‚0.6790217198431492 0loss/FBt4/assignment_loss‚0.3545661722487041 /loss/FRt1/detection_loss‚3.7232467770576476 /loss/FRt2/detection_loss‚2.6737414823388153 1loss/SRqqt2/detection_loss‚1.6219829856962353 /loss/FBt2/detection_loss‚1.7954235076904324 _step‚126 0loss/FRt3/assignment_loss‚3.5291284932646643 /loss/FRt4/detection_loss‚2.7288169326453375 2loss/SRqqt3/assignment_loss‚2.5947590624844588 % _timestamp‚1.7884085214915757e+09 /loss/FRt1/assignment_loss‚4.421841301236834 2loss/SRqqt2/assignment_loss‚1.7861810047905167 /loss/FBt1/assignment_loss‚0.303075541448834 0loss/SRqqt4/detection_loss‚2.897473917830558 0loss/FRt2/assignment_loss‚3.9409507153169163 1loss/SRqqt1/assignment_loss‚2.414380032004732 1loss/SRqqt1/detection_loss‚3.4412323480302636b`.\‡Ö 2loss/SRqqt3/assignment_loss‚2.5947590624844588 % _timestamp‚1.7884085214915757e+09 /loss/FRt1/assignment_loss‚4.421841301236834 2loss/SRqqt2/assignment_loss‚1.7861810047905167 /loss/FBt1/assignment_loss‚0.303075541448834 0loss/SRqqt4/detection_loss‚2.897473917830558 0loss/FRt2/assignment_loss‚3.9409507153169163 1loss/SRqqt1/assignment_loss‚2.414380032004732 1loss/SRqqt1/detection_loss‚3.4412323480302636 /loss/FBt1/detection_loss‚3.8353112898766994 /loss/FBt3/detection_loss‚1.2447936750059005 _runtime‚1131.903629837 /loss/FRt4/assignment_loss‚5.383458441701428 1loss/FBt3/assignment_loss‚0.27194852079146375 /loss/FRt3/detection_loss‚2.7587156850238177 1loss/SRqqt3/detection_loss‚1.8253111589346558 -loss/FBt4/detection_loss‚4.67413841382317 %loss/total_loss‚61.24410905555898 epoch‚1 trainer/global_step‚1549 1loss/SRqqt4/assignment_loss‚2.344609876473745 0loss/FBt2/assignment_loss‚0.6790217198431492 0loss/FBt4/assignment_loss‚0.3545661722487041 /loss/FRt1/detection_loss‚3.7232467770576476 /loss/FRt2/detection_loss‚2.6737414823388153 1loss/SRqqt2/detection_loss‚1.6219829856962353 /loss/FBt2/detection_loss‚1.7954235076904324 _step‚126 0loss/FRt3/assignment_loss‚3.5291284932646643 /loss/FRt4/detection_loss‚2.7288169326453375~wšòÞ‘ˆ‚(:† ÑåãÔÁ´Ç™ gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.102' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6204 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T3Ëœ‰‚(:… ÑåãÔ©½ disk.nvme0n1.in‚0.03125 disk.rbd2.out‚26.71826171875 disk.rbd3.out‚0 disk.sda.out‚ 46.53515625 disk.rbd1.out‚ 0.67578125 proc.memory.rssMB‚ 2879.8125 proc.cpu.threads‚17 network.sent‚ 2.389927e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.nvme0c0n1.in‚0.03125' disk.nvme0c0n1.out‚4499.25439453125 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 46.53515625) disk./.usagePercent‚70.82426213303675" disk.dm-0.out‚4499.25439453125 disk.sda2.in‚0 disk.drbd1777.in‚0% disk.nvme0n1.out‚4499.25439453125 disk.dm-0.in‚0.03125 disk.sda2.out‚0 disk.sda3.out‚ 46.53515625 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.dm-1.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0$ memory_percent‚99.94943141937256* proc.memory.percent‚17.576980590820312% disk./.usageGB‚1055.3326873779297 network.recv‚473350 disk.rbd3.in‚0& proc.memory.availableMB‚ 8.28515625 cpu‚25.33125271084926¾¤f뛊‚(: àåãԮ㿙 gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.39' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.677999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M·Q–‹‚(:€ àåãÔô¤Ê› disk.sda.out‚ 46.921875 disk.dm-1.out‚ 46.921875 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd1.out‚ 0.67578125 disk.nvme0n1.in‚0.03125 disk.sda.in‚0 disk.drbd1777.in‚0$ memory_percent‚99.95105266571045# proc.memory.rssMB‚ 2879.81640625 network.sent‚ 2.497513e+06 disk.dm-0.in‚0.03125 disk.dm-1.in‚0% disk./.usageGB‚1055.3328704833984 disk.rbd2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0& proc.memory.availableMB‚ 8.01953125* proc.memory.percent‚17.577004432678223 disk.rbd1.in‚0$ disk.nvme0n1.out‚4500.4580078125! disk.dm-0.out‚4500.458007812ïÙÊu5 network.recv‚479433) disk./.usagePercent‚70.82427442139839 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚0.03125 disk.rbd2.out‚26.71826171875 disk.sda2.in‚0 disk.sda3.out‚ 46.921875 disk.sda1.out‚0& disk.nvme0c0n1.out‚4500.4580078125 disk.sda3.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 cpu‚25.32754827152358ú· Æ2Œ- _wandbruntime‚1165  _runtime‚1165ÀDÂDÉà ' lr-AdamW/pg1‚0.00010071485370481405 ' lr-AdamW/pg2‚0.00010071485370481405 trainer/global_step‚1599 % _timestamp‚1.7884085554854445e+09 _runtime‚1165.897600879 _step‚127œh(™ÍŽÇ _runtime‚1165.897600879 _step‚127 ' lr-AdamW/pg1‚0.00010071485370481405 ' lr-AdamW/pg2‚0.00010071485370481405 trainer/global_step‚1599 % _timestamp‚1.7884085554854445e+09ŽÀ”Œ2- _wandbruntime‚1165  _runtime‚1165àÀÄ'¾O E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53165646 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2250929110016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20938023 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20938021 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚541 trainer/global_step‚1599 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚47083082599424 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚47082895597568 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53185684 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚47506930888704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 % _timestamp‚1.7884085554861808e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32227625 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20938023 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12220904 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9098591 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚ 1165.89848279 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚44832135486464 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2250760111104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚47083082599424 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚47506954828800 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44891029539215 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21319497 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216877056 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44891211699971 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 _step‚128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2250929110016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20938021 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚49797067596800 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2244445117187 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚44832135486464 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32227625 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚49796826779648 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚47082895597568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12220906 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚42646749543424 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚44832153489408 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240817152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2289895890944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚42646766582784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0…AiÝÉ'‘ÃO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21319497 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216877056 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44891211699971 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 _step‚128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2250929110016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20938021 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚49797067596800 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2244445117187 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚44832135486464 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32227625 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚49796826779648 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚47082895597568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12220906 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚42646749543424 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚44832153489408 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240817152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2289895890944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚42646766582784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53165646 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2250929110016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20938023 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20938021 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚541 trainer/global_step‚1599 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚47083082599424 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚47082895597568 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53185684 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚47506930888704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 % _timestamp‚1.7884085554861808e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32227625 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20938023 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12220904 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9098591 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚ 1165.89848279 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚44832135486464 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2250760111104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚47083082599424 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚47506954828800 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44891029539215 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000€ôÇ\2’- _wandbruntime‚1166  _runtime‚1166¡Î&“ÈM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53218156 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212669440 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚47111745115648 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚47111745115648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47535867777024 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚42672709765120 $ _timestamp‚1.788408556141098e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53198092 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21332035 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47535843836928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44859404851200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20950682 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53198092 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32247412 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9104289 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚44859422854144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2291313559552 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚42672692725760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53218156 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚49827394006016 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236609536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2245669160335 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12228283 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9103754 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚129 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2252322261504 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚47111558100480 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚49827157396480 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32267474 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44918361886095 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2252153249280 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2245834281939 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20950680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚49827394006016 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236609536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.smal(ô[ƒ!l_pool.freed‚ 2245669160335 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12228283 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9103754 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚129 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2252322261504 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚47111558100480 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚49827157396480 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32267474 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44918361886095 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2252153249280 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2245834281939 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20950680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53218156 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212669440 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚47111745115648 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚47111745115648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47535867777024 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚42672709765120 $ _timestamp‚1.788408556141098e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53198092 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21332035 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47535843836928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44859404851200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20950682 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53198092 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32247412 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9104289 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚44859422854144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2291313559552 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚42672692725760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53218156 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚46257660474423 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚51313398366720 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2361026458112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚51313628698112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48516035156992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚48516222158848 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21570971 % _timestamp‚1.7884085900942328e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54791298 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2320600432640 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚48516035156992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12589754 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚48952578299904 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54811336 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚551 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2360820066816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21570973 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21570973 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚46257842635179 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43943908660224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21570971 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚1649 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33240363 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33220327 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚46195603723264 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2313916935595 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206391296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230331392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2313751814199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚46257660474423 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚51313398366720 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2361026458112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚51313628698112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48516035156992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚48516222158848 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21570971 % _timestamp‚1.7884085900942328e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54791298 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2320600432640 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚48516035156992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12589754 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚48952578299904 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54811336 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚551 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2360820066816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21570973 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21570973 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚46257842635179 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43943908660224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21570971 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚1649 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33240363 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33220327 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚46195603723264 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2313916935595 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206391296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230331392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2313751814199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54843808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21583630 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12597131 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2315140978743 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33240114 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33260176 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚51343966336000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2315306100347 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21583632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234512384 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21974846 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚48981491248128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54823744 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _runtime‚1201.150723782 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2321993584128 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚46222873088000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21975382 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚46222891090944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2321824571904 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48544697659904 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚48544884675072 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2321993584128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚46222891090944Yä­Ó&¦ÍM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9378249 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _runtime‚1201.150723782 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2321993584128 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚46222873088000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21975382 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚46222891090944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2321824571904 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48544697659904 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚48544884675072 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2321993584128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚46222891090944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43969851842560 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51343731823616 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54843808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21583630 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12597131 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2315140978743 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33240114 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33260176 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚51343966336000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2315306100347 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21583632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234512384 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21974846 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚48981491248128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54823744 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34233065 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12958606 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216877056 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34233065 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34213029 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2390271755264 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22604978 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49949174716416 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚47559071960064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240817152 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚íyÏ+ê0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚47559071960064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22203923 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2383388754003 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47559089963008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2431909175808 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45241084816384 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚45241067777024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2390102756352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12958604 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 _step‚136 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚549 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22203921 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚47559089963008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47624473570387 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22203921 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚49949361718272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2390102756352 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 % _timestamp‚1.7884086244715502e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚49949174716416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56436988 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22604429 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚1234.883894931 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2390271755264 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2383223632607 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚1699 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2431692298752 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56416950 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚52830158827008 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9646372 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚47624291409631 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34213029 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚49949361718272 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚52829918009856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217àªõ#Ê'³ÄO XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚47559071960064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22203923 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2383388754003 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47559089963008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2431909175808 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45241084816384 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚45241067777024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2390102756352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12958604 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 _step‚136 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚549 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22203921 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚47559089963008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47624473570387 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22203921 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚49949361718272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2390102756352 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 % _timestamp‚1.7884086244715502e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚49949174716416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56436988 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22604429 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚1234.883894931 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2390271755264 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2383223632607 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚1699 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2431692298752 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56416950 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚52830158827008 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9646372 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚47624291409631 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34213029 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚49949361718272 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚52829918009856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9645825 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34233065 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12958606 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216877056 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34233065 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34213029 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2390271755264 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22604978 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49949174716416 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚47559071960064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240817152 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ˆ[Ø1Þ2´- _wandbruntime‚1235  _runtime‚1235^M´wÎ&µÈM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 221058048 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2433104533504 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12965981 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34232816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12965983 _step‚137 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2391664906752 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56449396 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47586341324800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2391664906752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚45267010959360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22617207 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22216582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34252878 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚47586341324800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22216580 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2391495894528 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34232816 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47651805917475 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚49977837219328 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2433325591552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56469460 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22216580 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚45267027998720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56449396 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚49978024234496 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9651226 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56469460 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244998144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50427162599424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2384612797151 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52860243192832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2384777918755 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22617738 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47651623756511Ï”) )¶ÍM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34232816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12965983 _step‚137 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2391664906752 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56449396 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47586341324800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2391664906752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚45267010959360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22617207 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22216582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34252878 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚47586341324800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22216580 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2391495894528 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34232816 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47651805917475 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚49977837219328 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2433325591552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56469460 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22216580 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚45267027998720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 C6Device÷î\ªStatsMonitor.on_train_batch_end/active.all.freed‚56449396 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚49978024234496 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9651226 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56469460 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244998144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50427162599424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2384612797151 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52860243192832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2384777918755 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22617738 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47651623756511 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 221058048 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2433104533504 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12965981 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160‰/¯¨”2·- _wandbruntime‚1235  _runtime‚1235g÷…ìW¸Ñ .loss/FBt1/detection_loss‚4.370845571712211 0loss/FRt2/assignment_loss‚3.6037651517174463 1loss/SRqqt3/assignment_loss‚2.406109241039857 /loss/FBt2/assignment_loss‚0.687999208766693 % _timestamp‚1.7884086251197042e+09 _step‚138 /loss/FRt1/assignment_loss‚4.548386118147108 2loss/SRqqt1/assignment_loss‚2.5838579403029547 1loss/FBt3/assignment_loss‚0.43723902480055316 1loss/SRqqt1/detection_loss‚3.7489014356224626 /loss/SRqqt3/detection_loss‚2.46912072631328 1loss/FBt4/assignment_loss‚0.19357290767657104 /loss/FRt2/detection_loss‚2.2084024078918225 /loss/FRt3/detection_loss‚2.9842652203263462 /loss/FBt2/detection_loss‚12.068569335280996 .loss/FBt4/detection_loss‚4.955276218056679 _runtime‚1235.531780743 2loss/SRqqt2/assignment_loss‚1.8598594151628323 /loss/FBt3/detection_loss‚1.3410510845821146 trainer/global_step‚1699 1loss/SRqqt4/assignment_loss‚1.968217167185574 /loss/FRt1/detection_loss‚2.8914426055219438 /loss/FRt4/detection_loss‚2.8589494228363037 /loss/FRt3/assignment_loss‚3.415029411380355 /loss/FRt4/assignment_loss‚4.881280275491568 1loss/SRqqt2/detection_loss‚1.7124754073910224 %loss/total_loss‚71.32634697198012 epoch‚1 1loss/FBt1/assignment_loss‚0.49656924711544653 0loss/SRqqt4/detection_loss‚2.635162427657988#¢4Ñ\¹Ö trainer/global_step‚1699 1loss/SRqqt4/assignment_loss‚1.968217167185574 /loss/FRt1/detection_loss‚2.8914426055219438 /loss/FRt4/detection_loss‚2.8589494228363037 /loss/FRt3/assignment_loss‚3.415029411380355 /loss/FRt4/assignment_loss‚4.881280275491568 1loss/SRqqt2/detection_loss‚1.7124754073910224 %loss/total_loss‚71.32634697198012 epoch‚1 1loss/FBt1/assignment_loss‚0.49656924711544653 0loss/SRqqt4/detection_loss‚2.635162427657988 .loss/FBt1/detection_loss‚4.370845571712211 0loss/FRt2/assignment_loss‚3.6037651517174463 1loss/SRqqt3/assignment_loss‚2.406109241039857 /loss/FBt2/assignment_loss‚0.687999208766693 % _timestamp‚1.7884086251197042e+09 _step‚138 /loss/FRt1/assignment_loss‚4.548386118147108 2loss/SRqqt1/assignment_loss‚2.5838579403029547 1loss/FBt3/assignment_loss‚0.43723902480055316 1loss/SRqqt1/detection_loss‚3.7489014356224626 /loss/SRqqt3/detection_loss‚2.46912072631328 1loss/FBt4/assignment_loss‚0.19357290767657104 /loss/FRt2/detection_loss‚2.2084024078918225 /loss/FRt3/detection_loss‚2.9842652203263462 /loss/FBt2/detection_loss‚12.068569335280996 .loss/FBt4/detection_loss‚4.955276218056679 _runtime‚1235.531780743 2loss/SRqqt2/assignment_loss‚1.8598594151628323 /loss/FBt3/detection_loss‚1.3410510845821146Š#Æíq‘º‚(:† ºæãÔ΢˜™ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.694' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9388 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06­xè »‚(:‚ ºæãÔÖå¾› network.recv‚508531 disk.rbd3.in‚0 disk.drbd1777.in‚0% disk.nvme0n1.out‚4611.42919921875 disk.rbd2.in‚0 disk.rbd2.out‚ 30.25390625 disk.dm-0.in‚0.03125& proc.memory.availableMB‚ 8.09765625 cpu‚25.308074404513633 disk.rbd3.out‚0$ memory_percent‚99.95057582855225* proc.memory.percent‚17.577004432678223) disk./.usagePercent‚70.83316684310023 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚4611.42919921875 disk.sda.out‚ 49.8671875 disk.drbd1777.out‚0 disk.rbd1.in‚0 network.sent‚ 2.736106e+06 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 49.8671875 disk.sda1.in‚0 disk.rbd1.out‚ 0.67578125 proc.cpu.threads‚17# disk./.usageGB‚1055.46537399292 disk.nvme0c0n1.in‚0.03125 disk.sda2.in‚0 disk.sda1.out‚0" disk.dm-0.out‚4611.42919921875 disk.dm-1.in‚0 disk.dm-1.out‚ 49.8671875 disk.sda.in‚0 disk.nvme0n1.in‚0.03125# proc.memory.rssMB‚ 2879.81640625£××O¼‚(:… ÉæãÔœŸ™ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚92.648' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5296 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÷v꽂(:„ ÉæãÔŽ®Øœ disk.drbd1777.out‚0 disk.rbd1.out‚ 0.67578125 disk.nvme0c0n1.in‚0.03125 disk.sda3.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚0.03125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 cpu‚25.305012069433054 disk.rbd2.out‚ 30.39453125 disk.rbd1.in‚0 disk.nvme0n1.in‚0.03125% disk.nvme0n1.out‚4710.50732421875$ memory_percent‚99.95214939117432& proc.memory.availableMB‚ 7.83984375 disk.rbd0.out‚0 network.sent‚ 2.844392e+06 disk.sda.in‚0 proc.cpu.threads‚17 network.recv‚515057 disk.sda2.out‚0 disk.sda3.out‚ 50.6953125 disk.sda.out‚ 50.6953125 disk.drbd1777.in‚0 disk.rbd3.in‚0) disk./.usagePercent‚70.84155723402742" disk.dm-0.out‚4710.50732421875' disk.nvme0c0n1.out‚4710.50732421875 disk.sda2.in‚0 disk.rbd2.in‚0% disk./.usageGB‚1055.5903968811035 disk.dm-1.out‚ 50.6953125# proc.memory.rssMB‚ 2879.81640625* proc.memory.percent‚17.577004432678223‚ïÃ2¾- _wandbruntime‚1269  _runtime‚1269xO%ÏÉ¿Ã ' lr-AdamW/pg1‚0.00010042202892648672 ' lr-AdamW/pg2‚0.00010042202892648672 trainer/global_step‚1749 % _timestamp‚1.7884086591492052e+09 _runtime‚1269.561376347 _step‚139uMRÀÎÀÈ trainer/global_step‚1749 % _timestamp‚1.7884086591492052e+09 _runtime‚1269.561376347 _step‚139 ' lr-AdamW/pg1‚0.00010042202892648672 ' lr-AdamW/pg2‚0.00010042202892648672‹–Z“V2Á- _wandbruntime‚1269  _runtime‚1269@¥(“Å'¿O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2459774078976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚48922558199808 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2459943077888 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2452860572411 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58042602 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚48922540196864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2502603446272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35205731 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51382314275840 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚48922540196864 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚51382314275840 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚559 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 _step‚140 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35225767 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚54346476568576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2452695451015 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22836871 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚51382501277696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚561 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226137088 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚51382501277696 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13327454 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚1269.562625199 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚51843897062400 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202196992 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35225767 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51382314275840 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚48922540196864 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚51382314275840 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚559 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 _step‚140 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35225767 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚54346476568576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2452695451015 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22836871 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚51382501277696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚561 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226137088 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚51382501277696 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13327454 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚1269.562625199 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚51843897062400 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202196992 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35225767 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2459774078976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚48922558199808 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2459943077888 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2452860572411 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58042602 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚48922540196864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2502603446272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35205731 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1749 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58075048 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚48949809561600 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2461167217152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13334833 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58095112 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚51411163793920 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚49018436852683 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2454249737163 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35225518 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚48949809561600 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58095112 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13334831 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚46564170076160 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2454084615559 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9924674 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2504231146496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2461167217152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚51872786070528 % _timestamp‚1.7884086597950897e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23259507 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚1270.207600785 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚48949827564544 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚54377041157120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚51411163793920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 _step‚141 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚49018254691719 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22849532 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22849530 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35245580 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚54376802450432 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚51410976778752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9924132 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22849530 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23258963 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚48949827564544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58075048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224Oµ—Ô&ÆÎM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 _step‚141 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚49018254691719 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22849532 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22849530 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35245580 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚54376802450432 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚51410976778752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9924132 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22849530 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23258963 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚48949827564544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58075048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚46564187115520 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22849532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚542 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2504016379904 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1749 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58075048 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚48949809561600 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2461167217152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13334833 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58095112 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚51411163793920 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚49018436852683 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2454249737163 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35225518 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚48949809561600 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58095112 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13334831 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚46564170076160 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2454084615559 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9924674 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2504231146496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2461167217152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚51872786070528 % _timestamp‚1.7884086597950897e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23259507 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚1270.207600785 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚48949827564544 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚54377041157120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚51411163793920ñSÄ2Ç- _wandbruntime‚1270  _runtime‚1270•ÃêVÈÐ 2loss/SRqqt1/assignment_loss‚2.4370606871027696 1loss/SRqqt2/detection_loss‚1.6913534995606716 %loss/total_loss‚58.97894103479986 0loss/FRt2/assignment_loss‚3.9069065581668507 1loss/SRqqt4/assignment_loss‚2.319801405948751 .loss/FRt4/detection_loss‚2.377767854723437 /loss/FBt2/detection_loss‚2.8809049257002286 .loss/FBt4/detection_loss‚4.630367432534695 $ _timestamp‚1.788408659796667e+09 /loss/FRt4/assignment_loss‚5.080100791207675 2loss/SRqqt3/assignment_loss‚2.5201976509655224 /loss/FRt3/assignment_loss‚3.434393729482378 1loss/FBt4/assignment_loss‚0.30375578562016015 trainer/global_step‚1749 1loss/FBt3/assignment_loss‚0.36419251423255145 /loss/FBt1/detection_loss‚2.9756559863686562 _runtime‚1270.208711177 _step‚142 /loss/FRt1/assignment_loss‚4.242003958958846 0loss/FBt2/assignment_loss‚0.6221414570028779 .loss/FRt1/detection_loss‚2.421679122516742 /loss/FRt2/detection_loss‚2.4745653017000717 1loss/SRqqt3/detection_loss‚2.3316668169755563 epoch‚1 2loss/SRqqt2/assignment_loss‚1.9568248391151428 1loss/SRqqt1/detection_loss‚2.6615045572582043 /loss/FBt3/detection_loss‚1.2708444955082365 .loss/FBt1/assignment_loss‚0.35418731809782 .loss/FRt3/detection_loss‚2.756628996559552 1loss/SRqqt4/detection_loss‚2.9644353494924656–ÔÕH[ÉÕ /loss/FRt1/assignment_loss‚4.242003958958846 0loss/FBt2/assignment_loss‚0.6221414570028779 .loss/FRt1/detection_loss‚2.421679122516742 /loss/FRt2/detection_loss‚2.4745653017000717 1loss/SRqqt3/detection_loss‚2.3316668169755563 epoch‚1 2loss/SRqqt2/assignment_loss‚1.9568248391151428 1loss/SRqqt1/detection_loss‚2.6615045572582043 /loss/FBt3/detection_loss‚1.2708444955082365 .loss/FBt1/assignment_loss‚0.35418731809782 .loss/FRt3/detection_loss‚2.756628996559552 1loss/SRqqt4/detection_loss‚2.9644353494924656 2loss/SRqqt1/assignment_loss‚2.4370606871027696 1loss/SRqqt2/detection_loss‚1.6913534995606716 %loss/total_loss‚58.97894103479986 0loss/FRt2/assignment_loss‚3.9069065581668507 1loss/SRqqt4/assignment_loss‚2.319801405948751 .loss/FRt4/detection_loss‚2.377767854723437 /loss/FBt2/detection_loss‚2.8809049257002286 .loss/FBt4/detection_loss‚4.630367432534695 $ _timestamp‚1.788408659796667e+09 /loss/FRt4/assignment_loss‚5.080100791207675 2loss/SRqqt3/assignment_loss‚2.5201976509655224 /loss/FRt3/assignment_loss‚3.434393729482378 1loss/FBt4/assignment_loss‚0.30375578562016015 trainer/global_step‚1749 1loss/FBt3/assignment_loss‚0.36419251423255145 /loss/FBt1/detection_loss‚2.9756559863686562 _runtime‚1270.208711177 _step‚142ŽÓþ͜ʂ(:‘ ØæãÔÓØ™ gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.363' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.672600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Mð»s Ë‚(: ØæãÔÑ‹…œ network.recv‚518715 disk.sda1.in‚0! disk.dm-0.out‚4811.2724609375 disk.nvme0c0n1.in‚0.03125$ disk./.usageGB‚1055.840316772461 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 0.76171875& disk.nvme0c0n1.out‚4811.2724609375$ memory_percent‚99.95224475860596& proc.memory.availableMB‚ 7.82421875 disk.sda3.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 30.39453125 disk.drbd1777.in‚0 disk.nvme0n1.in‚0.03125 disk.dm-1.in‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2879.81640625 proc.cpu.threads‚17 disk.sda3.out‚ 52.90234375$ disk.nvme0n1.out‚4811.2724609375 disk.dm-0.in‚0.03125 disk.dm-1.out‚ 52.90234375 disk.rbd3.out‚0) disk./.usagePercent‚70.85832956763318 disk.sda2.in‚0 disk.sda.out‚ 52.90234375 disk.rbd1.in‚0 disk.rbd2.in‚0* proc.memory.percent‚17.577004432678223 network.sent‚ 2.849687e+06 cpu‚25.3020948304401ñ5‹R‘Ì‚(:† çæãԻϴ™ gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.266' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0532 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÔÑÑëÍ‚(: çæãÔÐç㜠disk.dm-1.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 12.61328125# proc.memory.rssMB‚ 2879.81640625 disk.dm-1.out‚ 55.43359375 disk.rbd2.in‚0! disk.nvme0c0n1.in‚ 12.61328125 disk.drbd1777.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda1.in‚0$ memory_percent‚99.95088577270508 disk.dm-0.in‚ 12.61328125 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 55.43359375 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda3.out‚ 55.43359375 network.sent‚ 2.957378e+06 network.recv‚524548% disk./.usageGB‚1055.8390159606934 disk.rbd1.out‚ 0.82421875 disk.rbd2.out‚30.56298828125$ proc.memory.availableMB‚8.046875$ disk.nvme0n1.out‚4817.3388671875 disk.sda2.in‚0 cpu‚25.299189170085572 disk.rbd0.in‚0' disk./.usagePercent‚70.858242269064! disk.dm-0.out‚4817.3388671875& disk.nvme0c0n1.out‚4817.3388671875 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.sda.in‚0* proc.memory.percent‚17.577004432678223äSì2Î- _wandbruntime‚1303  _runtime‚1303™ÇlÆÏÀ _step‚143 & lr-AdamW/pg1‚0.0001003035912663008 & lr-AdamW/pg2‚0.0001003035912663008 trainer/global_step‚1799 $ _timestamp‚1.788408693549889e+09 _runtime‚1303.962118248C0’ËÐÅ & lr-AdamW/pg1‚0.0001003035912663008 & lr-AdamW/pg2‚0.0001003035912663008 trainer/global_step‚1799 $ _timestamp‚1.788408693549889e+09 _runtime‚1303.962118248 _step‚143|caÏ2Ñ- _wandbruntime‚1303  _runtime‚1303¤iØÒÁO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2529445401600 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36218469 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚50286026436608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23469821 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 trainer/global_step‚1799 _runtime‚1303.963145974 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59668254 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52815453835264 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚47835403049984 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚52815453835264 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚50286008433664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2529614400512 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59688292 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2522332390819 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚50357553280047 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚52815640837120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2529614400512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10191696 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚47835386010624 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2529445401600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13696306 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55863315990016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _step‚144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23469821 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚50357735440803 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682752 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2573558833664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23888002 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23469821 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 trainer/global_step‚1799 _runtime‚1303.963145974 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59668254 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52815453835264 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚47835403049984 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚52815453835264 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚50286008433664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2529614400512 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59688292 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2522332390819 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚50357553280047 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚52815640837120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2529614400512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10191696 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚47835386010624 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2529445401600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13696306 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55863315990016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _step‚144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23469821 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚50357735440803 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682752 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2573558833664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23888002 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚53318433481728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59720764 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 % _timestamp‚1.7884086941985939e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36238282 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36238282 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212669440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚50313277798400 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2531007552000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2531007552000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2530838539776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10196648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚47861346232320 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _step‚145 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23900329 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚1304.611490661 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚52844303353344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59700700 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2523556433967 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚53318457421824 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚1799 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2575188100096 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚50385067787891 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚50313295801344 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36218220 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236609536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23482480 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040FN²Ö&ÖÐM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59720764 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2530838539776 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚50313277798400 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59700700 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23900866 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚52844116338176 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚50313295801344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2523721555571 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23482482 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23482480 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚1304.611490661 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚52844303353344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59700700 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2523556433967 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚53318457421824 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚1799 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2575188100096 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚50385067787891 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚50313295801344 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36218220 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236609536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23482480 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚53318433481728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59720764 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 % _timestamp‚1.7884086941985939e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36238282 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36238282 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212669440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚50313277798400 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2531007552000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2531007552000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2530838539776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10196648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚47861346232320 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _step‚145 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23900329 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54248593394688 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51649476670464 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2599116724224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14065154 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51649494673408 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24531847 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2644427659776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 _step‚148 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10466691 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756Ì9ÝÊ'åÄO dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54735191884800 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 _step‚148 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10466691 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 # _timestamp‚1.78840872785439e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54248593394688 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51649476670464 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2599116724224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14065154 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51649494673408 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24531847 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2644427659776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚54277255897600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51676746035200 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24544112 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51751698723099 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37210922 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚54277442912768 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51676764038144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2600509862400 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37210922 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57409915329536 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚49158488309760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2593193373979 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2600509862400 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚550 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2600678874624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚49158505349120 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚54277442912768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 trainer/global_step‚1849 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24115432 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2645834436608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 247095296 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61346416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223155200 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24115432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51676746035200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14072531 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2593028252375 _step‚149 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10472129 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚548 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10471581 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24544662 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24115430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2646057591808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 % _timestamp‚1.7884087285034368e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24115430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚54277255897600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51676746035200 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24544112 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51751698723099 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37210922 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚54277442912768 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51676764038144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2600509862400 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37210922 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57409915329536 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚49158488309760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2593193373979 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2600509862400 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚550 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2600678874624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚49158505349120 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚54277442912768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 trainer/global_step‚1849 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24115432 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2645834436608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 247095296 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61346416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223155200 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24115432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51676746035200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14072531 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2593028252375 _step‚149 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10472129 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚548 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10471581 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24544662 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24115430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2646057591808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 % _timestamp‚1.7884087285034368e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24115430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚53012962910208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2661276027635 _runtime‚1372.928016299 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24735723 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38183837 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234525696 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55681919955968 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2668957045760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2715249255936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2668788046848 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55681919955968 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38203873 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚53012944907264 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38203873 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58896299137536 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24735721 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚53090815150463 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14434006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55681732954112 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚53012944907264 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24735723 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 % _timestamp‚1.7884087625156975e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2661110906239 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38183837 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2668788046848 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚53090997311219 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚53012962910208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25173108 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24735721 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10738582 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58896064611840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0›Üwo+õÆO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10739102 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62919558 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62939596 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚56180839296000 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚50429704244224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚56180815355904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55681919955968 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 tyH:¡"XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2668957045760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2715249255936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2668788046848 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55681919955968 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38203873 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚53012944907264 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38203873 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58896299137536 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24735721 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚53090815150463 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14434006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55681732954112 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚53012944907264 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24735723 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 % _timestamp‚1.7884087625156975e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2661110906239 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38183837 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2668788046848 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚53090997311219 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚53012962910208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25173108 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24735721 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10738582 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58896064611840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚53012962910208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2661276027635 _runtime‚1372.928016299 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24735723 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38183837 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234525696 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55710582472192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14441383 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38223686 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62952004 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58926396428288 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚53118147497343 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50455664465920 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50455647426560 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38203624 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53040232274944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2662500070783 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10744138 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚56209728304128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24748380 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2670350197248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38203624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24748380 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62972068 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚55710395457024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58926633037824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14441381 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2670350197248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚55710582472192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 _step‚153 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2670181185024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚1373.578865498 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53040214272000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25185519 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10744677 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2716668124160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2662665192387 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1899 % _timestamp‚1.7884087631661391e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24748382 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53040232274944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62972068 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38223686 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24748380 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2670350197248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38203624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24748380 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62972068 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚55710395457024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58926633037824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14441381 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2670350197248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚55710582472192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 _step‚153 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2670181185024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚1373.578865498 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53040214272000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25185519 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10744677 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2716668124160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2662665192387 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1899 % _timestamp‚1.7884087631661391e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24748382 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53040232274944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62972068 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38223686 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55710582472192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14441383 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38223686 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62952004 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58926396428288 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚53118147497343 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50455664465920 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50455647426560 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38203624 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53040232274944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2662500070783 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10744138™F .¢2ù- _wandbruntime‚1373  _runtime‚1373JoLiOúÉ 1loss/SRqqt1/assignment_loss‚2.297836434382659 1loss/SRqqt3/detection_loss‚2.3175834385885135 epoch‚1 /loss/FRt1/detection_loss‚3.5313693213051764 .loss/FRt2/detection_loss‚2.603329279551319 .loss/FBt2/detection_loss‚3.599030448360512 2loss/SRqqt2/assignment_loss‚2.0025971735778607 0loss/FBt1/assignment_loss‚0.4105945388675371 0loss/SRqqt1/detection_loss‚3.336881677691753 &loss/total_loss‚63.890174054348506 _runtime‚ 1373.58011967 /loss/FRt1/assignment_loss‚4.885404989637178 /loss/FRt3/assignment_loss‚3.376730345762693 0loss/FBt4/assignment_loss‚0.5222416553933726 trainer/global_step‚1899 % _timestamp‚1.7884087631680474e+09 _step‚154 /loss/FRt2/assignment_loss‚4.296079904425378 /loss/FRt4/assignment_loss‚4.546180323642843 0loss/SRqqt4/assignment_loss‚2.06160802145799 /loss/FBt3/detection_loss‚1.1268580628997589 /loss/FRt4/detection_loss‚2.5985880196094513 0loss/SRqqt2/detection_loss‚1.668603114391628 1loss/SRqqt4/detection_loss‚2.2279344058699078 1loss/SRqqt3/assignment_loss‚2.285278262032403 0loss/FBt2/assignment_loss‚0.6911598103380518 0loss/FBt3/assignment_loss‚0.3318599294253213 .loss/FRt3/detection_loss‚2.639508063212419 -loss/FBt1/detection_loss‚4.95032554932616 .loss/FBt4/detection_loss‚5.582591284598623aÈ_6TûÎ trainer/global_step‚1899 % _timestamp‚1.7884087631680474e+09 _step‚154 /loss/FRt2/assignment_loss‚4.296079904425378 /loss/FRt4/assignment_loss‚4.546180323642843 0loss/SRqqt4/assignment_loss‚2.06160802145799 /loss/FBt3/detection_loss‚1.1268580628997589 /loss/FRt4/detection_loss‚2.5985880196094513 0loss/SRqqt2/detection_loss‚1.668603114391628 1loss/SRqqt4/detection_loss‚2.2279344058699078 1loss/SRqqt3/assignment_loss‚2.285278262032403 0loss/FBt2/assignment_loss‚0.6911598103380518 0loss/FBt3/assignment_loss‚0.3318599294253213 .loss/FRt3/detection_loss‚2.639508063212419 -loss/FBt1/detection_loss‚4.95032554932616 .loss/FBt4/detection_loss‚5.582591284598623 1loss/SRqqt1/assignment_loss‚2.297836434382659 1loss/SRqqt3/detection_loss‚2.3175834385885135 epoch‚1 /loss/FRt1/detection_loss‚3.5313693213051764 .loss/FRt2/detection_loss‚2.603329279551319 .loss/FBt2/detection_loss‚3.599030448360512 2loss/SRqqt2/assignment_loss‚2.0025971735778607 0loss/FBt1/assignment_loss‚0.4105945388675371 0loss/SRqqt1/detection_loss‚3.336881677691753 &loss/total_loss‚63.890174054348506 _runtime‚ 1373.58011967 /loss/FRt1/assignment_loss‚4.885404989637178 /loss/FRt3/assignment_loss‚3.376730345762693 0loss/FBt4/assignment_loss‚0.5222416553933726šÌ(M‡ü‚(:… ÁçãÔï€Í™ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚87.303' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.4606 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+DFâý‚(: ÁçãÔÎÿ®œ network.recv‚555997 disk.nvme0n1.in‚ 422.44140625& disk.nvme0c0n1.out‚6096.0380859375 disk.sda2.out‚0 disk.drbd1777.out‚0$ disk.nvme0n1.out‚6096.0380859375 disk.rbd2.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 422.44140625 disk.sda3.in‚0 proc.memory.availableMB‚7.25 network.sent‚ 3.197619e+06 disk.sda1.out‚0 disk.dm-0.in‚ 422.44140625 disk.rbd2.out‚ 32.3955078125 disk.sda3.out‚ 62.16796875$ memory_percent‚99.95574951171875 disk.drbd1777.in‚0 disk.rbd1.out‚ 0.982421875 disk.dm-1.out‚ 62.16796875 disk.rbd3.in‚0 disk.sda.in‚0 disk.rbd1.in‚0! disk.dm-0.out‚6096.0380859375 disk.âÂ9dm-1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.577004432678223% disk./.usageGB‚1056.9099617004395 disk.sda.out‚ 62.16796875 disk.rbd0.out‚0 disk.sda2.in‚0) disk./.usagePercent‚70.93011433624169 disk.sda1.in‚0# proc.memory.rssMB‚ 2879.81640625 cpu‚25.2830600752551363¶¬‘þ‚(:† ÐçãÔºž™ gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.999' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0È7lyÿ‚(:“ ÐçãÔÙú“œ disk.dm-0.in‚ 422.44140625 disk.dm-1.in‚0 disk.dm-1.out‚ 62.8359375 disk.rbd0.out‚0 disk.sda.out‚ 62.8359375# proc.memory.rssMB‚ 2879.81640625" disk.nvme0c0n1.in‚ 422.44140625 disk.sda2.in‚0 network.sent‚ 3.305094e+06 network.recv‚561698) disk./.usagePercent‚70.93014556916086& disk.nvme0c0n1.out‚6204.5048828125 disk.sda1.in‚0& proc.memory.availableMB‚ 7.41796875 cpu‚25.280429038760335 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.982421875 disk.nvme0n1.in‚ 422.44140625! disk.dm-0.out‚6204.5048828125 disk.rbd3.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 62.8359375 disk.sda.in‚0% disk./.usageGB‚1056.9104270935059$ disk.nvme0n1.out‚6204.5048828125 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.drbd1777.in‚0* proc.memory.percent‚17.577004432678223$ memory_percent‚99.95472431182861 disk.rbd2.out‚ 32.3955078125 disk.sda3.in‚0 disk.drbd1777.out‚0 proc.cpu.threads‚17Ìmm/2€- _wandbruntime‚1408  _runtime‚1408l€HþÅ¿ trainer/global_step‚1949 % _timestamp‚1.7884087976224885e+09 _runtime‚1408.034649224 _step‚155 % lr-AdamW/pg1‚9.98861355987397e-05 % lr-AdamW/pg2‚9.98861355987397e-05Í­QÁÊ‚Ä % lr-AdamW/pg2‚9.98861355987397e-05 trainer/global_step‚1949 % _timestamp‚1.7884087976224885e+09 _runtime‚1408.034649224 _step‚155 % lr-AdamW/pg1‚9.98861355987397e-05›¸ôe2ƒ- _wandbruntime‚1408  _runtime‚1408Â[gÅ'„¿O 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚57626486707200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚60412599561216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2730747846043 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57114872513536 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2738459369472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2738459369472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25815121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64545210 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚ 1408.0355812 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚54376413144064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2786136794112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 % _timestamp‚1.7884087976232073e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39196575 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39176539 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11012783 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25368673 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14802856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230331392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25368673 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚57626462767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11012267 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 trainer/global_step‚1949 _step‚156 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57115059515392 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2786343185408 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2730582724647 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25368671 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚516 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚51726863361024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57114872513536 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64565248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39196575 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57115059515392 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 IãiÊ'…ÄO IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚60412599561216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2730747846043 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57114872513536 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2738459369472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2738459369472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25815121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64545210 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚ 1408.0355812 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚54376413144064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2786136794112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 % _timestamp‚1.7884087976232073e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39196575 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39176539 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11012783 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25368673 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14802856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230331392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25368673 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚57626462767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11012267 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 trainer/global_step‚1949 _step‚156 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57115059515392 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2786343185408 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2730582724647 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25368671 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚516 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚51726863361024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57114872513536 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64565248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39196575 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57115059515392 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038œ$ƒ_º2†- _wandbruntime‚1408  _runtime‚1408Ž+×xÑ&‡ËM ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2731971889191 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚552 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25827886 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25381330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14810233 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚54403682508800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39216388 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚57655375715328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2787553550848 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚54403682508800 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25828438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 trainer/global_step‚1949 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚54403700511744 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25381330 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚57143722031616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚60443165875712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚57655399655424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64597720 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39196326 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11018205 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14810231 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64597720 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 % _timestamp‚1.7884087982930365e+09 _runtime‚1408.705934617 _step‚157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11017655 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2740021519872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25381332 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212669440 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64577656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236609536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0£¢?ˆÐM REDeviceStatsMonitor.on_train_batch_end/inactive_split.ñÏu—&small_pool.freed‚11017655 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2740021519872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25381332 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212669440 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64577656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236609536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2731971889191 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚552 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25827886 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25381330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14810233 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚54403682508800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39216388 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚57655375715328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2787553550848 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚54403682508800 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25828438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 trainer/global_step‚1949 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚54403700511744 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25381330 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚57143722031616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚60443165875712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚57655399655424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64597720 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39196326 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11018205 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14810231 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64597720 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 % _timestamp‚1.7884087982930365e+09 _runtime‚1408.705934617 _step‚157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2Á/Ò2‰- _wandbruntime‚1408  _runtime‚1408‘1†VŠÐ trainer/global_step‚1949 % _timestamp‚1.7884087982951767e+09 1loss/SRqqt4/assignment_loss‚2.099609521158198 1loss/FBt1/assignment_loss‚0.35141230700307796 0loss/SRqqt1/detection_loss‚2.965601474046707 1loss/SRqqt4/detection_loss‚2.1970547513758882 .loss/FBt4/detection_loss‚4.636865707114339 %loss/total_loss‚60.74144309609903 1loss/FBt4/assignment_loss‚0.40399688398338185 /loss/FRt1/detection_loss‚2.8325849962540164 /loss/FRt2/detection_loss‚1.9958177152751149 /loss/FRt4/detection_loss‚3.1324542130742756 .loss/FBt1/detection_loss‚5.556667806286561 0loss/FRt2/assignment_loss‚3.4040524173473967 1loss/SRqqt3/assignment_loss‚2.566705909310555 /loss/FBt3/assignment_loss‚0.335208439507159 0loss/FBt2/assignment_loss‚0.4293093677196238 epoch‚1 /loss/FRt4/assignment_loss‚4.030518668038504 2loss/SRqqt2/assignment_loss‚1.5905825060022913 .loss/FRt3/detection_loss‚2.460552067919211 /loss/FBt3/detection_loss‚1.7976694540070617 _runtime‚1408.707214157 0loss/FRt3/assignment_loss‚3.1161686696789483 2loss/SRqqt1/assignment_loss‚2.9820924282073973 /loss/FBt2/detection_loss‚3.9442636171976724 _step‚158 /loss/FRt1/assignment_loss‚4.454544932414324 0loss/SRqqt2/detection_loss‚1.280094244800832 0loss/SRqqt3/detection_loss‚2.177614998376491:{îÇ[‹Õ 2loss/SRqqt1/assignment_loss‚2.9820924282073973 /loss/FBt2/detection_loss‚3.9442636171976724 _step‚158 /loss/FRt1/assignment_loss‚4.454544932414324 0loss/SRqqt2/detection_loss‚1.280094244800832 0loss/SRqqt3/detection_loss‚2.177614998376491 trainer/global_step‚1949 % _timestamp‚1.7884087982951767e+09 1loss/SRqqt4/assignment_loss‚2.099609521158198 1loss/FBt1/assignment_loss‚0.35141230700307796 0loss/SRqqt1/detection_loss‚2.965601474046707 1loss/SRqqt4/detection_loss‚2.1970547513758882 .loss/FBt4/detection_loss‚4.636865707114339 %loss/total_loss‚60.74144309609903 1loss/FBt4/assignment_loss‚0.40399688398338185 /loss/FRt1/detection_loss‚2.8325849962540164 /loss/FRt2/detection_loss‚1.9958177152751149 /loss/FRt4/detection_loss‚3.1324542130742756 .loss/FBt1/detection_loss‚5.556667806286561 0loss/FRt2/assignment_loss‚3.4040524173473967 1loss/SRqqt3/assignment_loss‚2.566705909310555 /loss/FBt3/assignment_loss‚0.335208439507159 0loss/FBt2/assignment_loss‚0.4293093677196238 epoch‚1 /loss/FRt4/assignment_loss‚4.030518668038504 2loss/SRqqt2/assignment_loss‚1.5905825060022913 .loss/FRt3/detection_loss‚2.460552067919211 /loss/FBt3/detection_loss‚1.7976694540070617 _runtime‚1408.707214157 0loss/FRt3/assignment_loss‚3.1161686696789483žðº¦ô‘Œ‚(:† ßçãÔÕ’§™ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.557' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1114 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M4ƒ‚(:” ßçãÔÊ›³œ disk.drbd1777.out‚0* proc.memory.percent‚17.577004432678223 disk.rbd0.out‚0 disk.sda3.in‚0 disk.rbd1.out‚ 0.982421875$ memory_percent‚99.95641708374023 disk.dm-1.in‚0 disk.dm-1.out‚ 63.54296875 disk.rbd2.in‚0 disk.rbd1.in‚0 network.sent‚ 3.310781e+06% disk./.usageGB‚1056.9104499816895% disk.nvme0n1.out‚6204.87646484375! disk.nvme0c0n1.in‚ 425.0078125 disk.drbd1777.in‚0# proc.memory.rssMB‚ 2879.81640625) disk./.usagePercent‚70.93014710520607 disk.rbd0.in‚0 disk.sda.out‚ 63.54296875 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda.in‚0$ proc.memory.availableMB‚7.140625 cpu‚25.278010860363505 proc.cpu.threads‚17 network.recv‚565572 disk.rbd2.out‚ 34.1103515625 disk.rbd3.out‚0 disk.dm-0.in‚ 425.0078125" disk.dm-0.out‚6204.87646484375 disk.sda1.in‚0 disk.nvme0n1.in‚ 425.0078125 disk.rbd3.in‚0' disk.nvme0c0n1.out‚6204.87646484375 disk.sda2.out‚0 disk.sda3.out‚ 63.54296875Ò8t‘Ž‚(:† îçãÔÏ®¹™ gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.304' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8608 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™úz‚(:‘ îçãÔ¶ disk.rbd3.out‚0' disk.nvme0c0n1.out‚6304.22314453125 disk.sda3.out‚ 63.9453125 disk.sda1.in‚0$ proc.memory.availableMB‚7.515625 cpu‚25.275621612136884) disk./.usagePercent‚70.94692481497005 disk.rbd3.in‚0 disk.dm-1.in‚0$ memory_percent‚99.95412826538086 network.recv‚572890" disk.dm-0.out‚6304.22314453125 network.sent‚ 3.419567e+06 disk.rbd2.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.982421875 disk.dm-1.out‚ 63.9453125* proc.memory.percent‚17.577004432678223! disk.nvme0c0n1.in‚ 425.0078125 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 425.0078125 proc.cpu.threads‚17 disk.rbd2.out‚ 34.1103515625 disk.drbd1777.in‚0% disk.nvme0n1.out‚6304.22314453125% disk./.usageGB‚1057.1604499816895 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda.out‚ 63.9453125 disk.nvme0n1.in‚ 425.0078125 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2879.81640625W|¬i‘‚(:† ýçãÔ«±¶™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.737' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3474 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N*‡ ‘‚(:• ýçãÔˆà‹ disk.rbd1.out‚ 0.982421875" disk.dm-0.out‚6304.57763671875 disk.rbd3.out‚0# proc.memory.rssMB‚ 2879.81640625 disk.rbd2.in‚0 disk.rbd2.out‚34.26318359375 disk.sda2.in‚0 cpu‚25.27348635289186 proc.cpu.threads‚17' disk.nvme0c0n1.out‚6304.57763671875 disk.sda2.out‚0 disk.sda.in‚0 disk.nvme0n1.in‚ 425.0078125 disk.rbd3.in‚0 disk.sda.out‚ 64.70703125 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0% disk./.usageGB‚1057.1650657653809! disk.nvme0c0n1.in‚ 425.0078125 disk.rbd0.out‚0$ memory_percent‚99.95732307434082% proc.memory.availableMB‚ 6.9921875 network.recv‚576908) disk./.usagePercent‚70.94723458408647 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 425.0078125 disk.dm-1.out‚ 64.70703125 network.sent‚ 3.425635e+06 disk.sda3.out‚ 64.70703125 disk.sda1.out‚0% disk.nvme0n1.out‚6304.57763671875* proc.memory.percent‚17.577004432678223™j&¡2’- _wandbruntime‚1443  _runtime‚1443©ÅÈãÇ“Á & lr-AdamW/pg1‚9.972636614470687e-05 & lr-AdamW/pg2‚9.972636614470687e-05 trainer/global_step‚1999 % _timestamp‚1.7884088334754608e+09 _runtime‚1443.887649379 _step‚159~hn7Ì”Æ _step‚159 & lr-AdamW/pg1‚9.972636614470687e-05 & lr-AdamW/pg2‚9.972636614470687e-05 trainer/global_step‚1999 % _timestamp‚1.7884088334754608e+09 _runtime‚1443.887649379Ÿb«L»2•- _wandbruntime‚1443  _runtime‚1443„ù[ïÇ'–ÁO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2800219664451 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚58548012072960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚558 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2808299691008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 _step‚160 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚556 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚58548199074816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 trainer/global_step‚1999 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66170862 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40189277 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚58548199074816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26456628 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚55739899383808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚53024022477824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26001623 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2808130692096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622848 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40169241 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15171706 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2808299691008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚58548012072960 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚61929423648256 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚59072110178304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2857076847104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26457186 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0Uù'‚J—ÆO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622848 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40169241 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15171706 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2808299691008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚58548012072960 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚61929423648256 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚59072110178304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2857076847104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26457186 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2800219664451 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚58548012072960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚558 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2808299691008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 _step‚160 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚556 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚58548199074816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 trainer/global_step‚1999 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66170862 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40189277 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚58548199074816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26456628 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚55739899383808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚53024022477824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26001623 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2808130692096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 Ðl º2˜- _wandbruntime‚1444  _runtime‚1444ÇÅDÑ&™ËM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232415232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚55767168748544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15179081 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2858702618624 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚53049965660160 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2801443707599 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2801608829203 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26014282 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚59101047066624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26469442 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15179083 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11290361 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚58576861591040 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66223372 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66223372 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40209090 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40189028 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 _step‚161 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26014280 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26469979 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚55851591528723 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26014280 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40189028 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26014282 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2809692842496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2858494143488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11290896 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚1444.548244955 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚61959517270016 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2809523830272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2809692842496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40209090 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66223372 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66223372 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40209090 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40189028 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 _step‚161 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26014280 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26469979 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚55851591528723 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26014280 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40189028 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26014282 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2809692842496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2858494143488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11290896 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚1444.548244955 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚61959517270016 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2809523830272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2809692842496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40209090 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚55767168748544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15179081 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2858702618624 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚53049965660160 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2801443707599 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2801608829203 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26014282 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚59101047066624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26469442 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15179083 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11290361 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0¡¤¹ð2›- _wandbruntime‚1444  _runtime‚1444ÀýXœÒ 0loss/FBt4/assignment_loss‚0.4833439112268252 /loss/FBt2/detection_loss‚2.8021347522738917 _runtime‚1444.549593025 /loss/FRt1/assignment_loss‚4.988676315262204 2loss/SRqqt4/assignment_loss‚2.1338402535021306 /loss/FRt4/assignment_loss‚4.033030693871634 0loss/SRqqt4/detection_loss‚2.037574979662895 .loss/FBt1/detection_loss‚6.648317478597164 % _timestamp‚1.7884088341374753e+09 _step‚162 /loss/FRt2/assignment_loss‚3.722515247084878 0loss/FRt3/assignment_loss‚2.9380149993785594 2loss/SRqqt3/assignment_loss‚2.2241327356208456 1loss/FBt3/assignment_loss‚0.25520388208116385 .loss/FRt3/detection_loss‚2.474229137911353 2loss/SRqqt2/assignment_loss‚1.9028940147930575 /loss/FBt2/assignment_loss‚0.451411971822381 /loss/FBt4/detection_loss‚4.3373010493814945 epoch‚1 /loss/FRt2/detection_loss‚2.2942769917574797 /loss/FRt1/detection_loss‚2.7186169305018018 .loss/FRt4/detection_loss‚2.660001196180071 1loss/SRqqt1/detection_loss‚3.0670857450791766 1loss/SRqqt2/detection_loss‚1.9983683902650111 1loss/SRqqt3/detection_loss‚1.9594764686274257 /loss/FBt3/detection_loss‚1.6185479912886402 &loss/total_loss‚60.729283136181564 trainer/global_step‚1999 1loss/SRqqt1/assignment_loss‚2.644970649055072 0loss/FBt1/assignment_loss‚0.3353173509564158λ{0]× 0loss/SRqqt4/detection_loss‚2.037574979662895 .loss/FBt1/detection_loss‚6.648317478597164 % _timestamp‚1.7884088341374753e+09 _step‚162 /loss/FRt2/assignment_loss‚3.722515247084878 0loss/FRt3/assignment_loss‚2.9380149993785594 2loss/SRqqt3/assignment_loss‚2.2241327356208456 1loss/FBt3/assignment_loss‚0.25520388208116385 .loss/FRt3/detection_loss‚2.474229137911353 2loss/SRqqt2/assignment_loss‚1.9028940147930575 /loss/FBt2/assignment_loss‚0.451411971822381 /loss/FBt4/detection_loss‚4.3373010493814945 epoch‚1 /loss/FRt2/detection_loss‚2.2942769917574797 /loss/FRt1/detection_loss‚2.7186169305018018 .loss/FRt4/detection_loss‚2.660001196180071 1loss/SRqqt1/detection_loss‚3.0670857450791766 1loss/SRqqt2/detection_loss‚1.9983683902650111 1loss/SRqqt3/detection_loss‚1.9594764686274257 /loss/FBt3/detection_loss‚1.6185479912886402 &loss/total_loss‚60.729283136181564 trainer/global_step‚1999 1loss/SRqqt1/assignment_loss‚2.644970649055072 0loss/FBt1/assignment_loss‚0.3353173509564158 0loss/FBt4/assignment_loss‚0.4833439112268252 /loss/FBt2/detection_loss‚2.8021347522738917 _runtime‚1444.549593025 /loss/FRt1/assignment_loss‚4.988676315262204 2loss/SRqqt4/assignment_loss‚2.1338402535021306 /loss/FRt4/assignment_loss‚4.033030693871634¢\?¸ÿ‘ž‚(:† ŒèãÔ˶ٙ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.557' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1114 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pŸ‚(:“ ŒèãÔæ·œ disk.rbd1.out‚ 0.982421875" disk.nvme0c0n1.in‚ 432.80078125 disk.drbd1777.in‚0 disk.rbd2.out‚34.26318359375 disk.sda3.in‚0 disk.rbd1.in‚0* proc.memory.percent‚17.577028274536133 disk.sda.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 432.80078125 disk.dm-0.out‚6408.712890625 cpu‚25.271243960790216 proc.cpu.threads‚17# disk.nvme0n1.out‚6408.712890625 network.sent‚ 3.430944e+06) disk./.usagePercent‚70.94722485580017 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda.out‚ 65.27734375 disk.sda1.in‚0 network.recv‚580653 disk.rbd3.in‚0 disk.sda2.in‚0 disk.drbd1777.out‚0$ memory_percent‚99.95591640472412& proc.memory.availableMB‚ 7.22265625" proc.memory.rssMB‚ 2879.8203125 disk.dm-1.in‚0% disk.nvme0c0n1.out‚6408.712890625 disk.nvme0n1.in‚ 432.80078125% disk./.usageGB‚1057.1649208068848 disk.dm-1.out‚ 65.27734375 disk.sda3.out‚ 65.277343752V⺑ ‚(:† ›èãÔ©õÝš gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚110.543' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1086 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ҷ¡‚(:’ ›èãÔ¥˜ûœ disk.drbd1777.out‚0 network.sent‚ 3.538654e+06 network.recv‚586266$ disk./.usageGB‚1057.164939880371 disk.sda.out‚ 65.94921875 disk.sda1.in‚0 disk.dm-0.in‚ 432.80078125 disk.sda1.out‚0 disk.dm-1.out‚ 65.94921875 disk.rbd0.out‚0 disk.rbd2.out‚34.41943359375 disk.sda2.out‚0$ memory_percent‚99.95615482330322* proc.memory.percent‚17.577028274536133 disk.sda3.in‚0 disk.sda.in‚0# disk.nvme0n1.out‚6507.830078125% disk.nvme0c0n1.out‚6507.830078125) disk./.usagePercent‚70.94722613583785 disk.rbd1.in‚0 disk.rbd3.in‚0 cpu‚25.269172501313605 disk.rbd1.out‚ 0.982421875 disk.nvme0n1.in‚ 432.80078125 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 65.94921875 disk.rbd3.out‚0& proc.memory.availableMB‚ 7.18359375 disk.drbd1777.in‚0 disk.dm-0.out‚6507.830078125 disk.rbd2.in‚0" proc.memory.rssMB‚ 2879.8203125 proc.cpu.threads‚17 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 432.80078125_d42¢- _wandbruntime‚1479  _runtime‚1479.C:Æ£À & lr-AdamW/pg1‚9.955634161329244e-05 & lr-AdamW/pg2‚9.955634161329244e-05 trainer/global_step‚2049 % _timestamp‚1.7884088690191903e+09 _runtime‚ 1479.43135187 _step‚163eè©0ˤŠ% _timestamp‚1.7884088690191903e+09 _runtime‚ 1479.43135187 _step‚163 & lr-AdamW/pg1‚9.955634161329244e-05 & lr-AdamW/pg2‚9.955634161329244e-05 trainer/global_step‚2049£¤Q.2¥- _wandbruntime‚1479  _runtime‚1479(e‹Íü¦ÁO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2877802014720 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2877971013632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15540556 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41161943 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682752 @8DeviceStatN¼ÝË"sMonitor.on_train_batch_start/segment.all.peak‚416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚57103367620608 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59981338634240 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚57103367620608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2927983226368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41161943 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622848 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2928195909120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚57190890116843 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15540554 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2877971013632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2869691482859 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚57190707956087 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26634571 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59981151632384 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚57103349617664 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54321198633984 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7884088690199275e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚63445740815872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27099557 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60517757589504 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59981151632384 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚57103349617664 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54321198633984 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7884088690199275e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚63445740815872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27099557 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60517757589504 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41161943 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622848 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2928195909120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚57190890116843 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15540554 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2877971013632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2869691482859 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚57190707956087 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0¤—c¯2¨- _wandbruntime‚1480  _runtime‚1480BvëÑ&©ËM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67828960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2879364165120 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚57130636985344 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67849024 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2929613488128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67828960 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚57130618982400 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27112104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41181730 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15547931 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15547933 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2879364165120 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26647230 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11564703 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884088696873498e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26647232 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚60009814135296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚60546670537728 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚165 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚57130618982400 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532d?ÆYCªÐM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67849024 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26647232 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 _runtime‚1480.099903827 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27112636 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚57130636985344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚530 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11564173 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚54347124776960 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚57218040302967 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚165 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚57130618982400 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67828960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2879364165120¢øϯ“ ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚57130636985344 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67849024 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2929613488128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67828960 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚57130618982400 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27112104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41181730 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15547931 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15547933 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2879364165120 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26647230 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11564703 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884088696873498e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26647232 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚60009814135296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚60546670537728 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2999147110400 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2947642336256 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2947473337344 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27267521 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15909406 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚55618357750784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2998949107712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 % _timestamp‚1.7884089044831445e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61414291191808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 _runtime‚1514.895585218 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚64962354108416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2938998179871 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚58466817854464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11830499 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚58557521052051 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ÙÊÂd'·ÆO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚58466835857408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42174681 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27740420 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64962576051200 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69442204 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚61414478193664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58466835857408 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2947642336256 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚61963428940800 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _step‚168 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42154645 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2999147110400 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2947642336256 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2947473337344 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27267521 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15909406 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚55618357750784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2998949107712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 % _timestamp‚1.7884089044831445e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61414291191808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 _runtime‚1514.895585218 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚64962354108416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2938998179871 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚58466817854464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11830499 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚58557521052051 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27267523 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 trainer/global_step‚2099 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚61414478193664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27267521 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15909404 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27752878 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69474676 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69454612 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27280180 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42174432 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206377984 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61442953694720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230318080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58584853399139 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚58584671238175 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27280180 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69474676 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚61443140709888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27280182 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11836097 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64992917391872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3000575502848 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 trainer/global_step‚2099 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42174432 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚61992341889024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚61992317948928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7884089051390429e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚58494087219200 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2949035487744 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42194494 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚55644283893760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚58494087219200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11836596 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27752878 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69474676 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69454612 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27280180 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42174432 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206377984 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61442953694720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230318080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58584853399139 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚58584671238175 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27280180 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69474676 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚61443140709888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27280182 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11836097 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64992917391872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3000575502848 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 trainer/global_step‚2099 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42174432 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚61992341889024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚61992317948928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7884089051390429e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚58494087219200 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2949035487744 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42194494 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚55644283893760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚58494087219200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11836596 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚62847617753088 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚62847617753088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3008469998279 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚59830304094208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3070126050304 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3017144659968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚63409052411904 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚62847430751232 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚172 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71067856 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214779904 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59924151987259 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 % _timestamp‚1.7884089395964758e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28381458 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚66478963682304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27900473 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238720000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43167383 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚59830286091264 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12103742 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43167383 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚59830304094208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3017313658880 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63409076352000 trainer/global_step‚2149 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16278256 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3008635119675 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27900471 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3069911270400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71047818 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62847430751232 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3017144659968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚56915499828224 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56915516867584 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12103204 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756V!ðAÌ'ÉÆO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27900473 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚62847430751232 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚172 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71067856 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214779904 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59924151987259 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 % _timestamp‚1.7884089395964758e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28381458 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚66478963682304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27900473 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238720000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43167383 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚59830286091264 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12103742 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43167383 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚59830304094208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3017313658880 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63409076352000 trainer/global_step‚2149 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16278256 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3008635119675 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27900471 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3069911270400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71047818 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62847430751232 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3017144659968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚56915499828224 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56915516867584 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12103204 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _runtime‚1550.008760168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28381998 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚62847617753088 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚62847617753088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3008469998279 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚59830304094208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3070126050304 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3017144659968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚63409052411904 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚63437989300224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208475136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43187196 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚66509296835072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚63437965360128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3009859162823 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚1550.655180617 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59857555456000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56941443010560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43187196 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71080264 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3071331474944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _step‚173 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71100328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12109202 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3018537798144 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚62876093254144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27913130 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3071539950080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3010024284427 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28394835 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59951484334347 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 trainer/global_step‚2149 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27913132 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43167134 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚549 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59951302173383 $ _timestamp‚1.788408940242455e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3018706810368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27913130 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27913132 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚551 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚62876280269312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚59857573458944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43167134 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚59857555456000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28394284 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56941460049920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3018537798144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0Sõø(jÌÏM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28394835 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59951484334347 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 trainer/global_step‚2149 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27913132 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43167134 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚549 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59951302173383 $ _timestamp‚1.788408940242455e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3018706810368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27913130 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27913132 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚551 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚62876280269312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚59857573458944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43167134 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚59857555456000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28394284 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56941460049920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3018537798144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59857573458944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚63437989300224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208475136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 E?DeviceStatsMonitor.on_traiÚ \k n_batch_end/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43187196 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚66509296835072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚63437965360128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3009859162823 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚1550.655180617 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59857555456000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56941443010560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43187196 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71080264 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3071331474944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _step‚173 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71100328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12109202 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3018537798144 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚62876093254144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27913130 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3071539950080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3010024284427 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408­nÁó+2Í- _wandbruntime‚1550  _runtime‚1550z+…VVÎÐ 1loss/SRqqt4/assignment_loss‚2.217841077595949 /loss/FRt4/detection_loss‚3.0310828701957413 1loss/SRqqt3/detection_loss‚1.9745993782173503 /loss/FBt2/detection_loss‚2.3525409251700475 .loss/FBt3/detection_loss‚1.352991123511562 _step‚174 /loss/FRt2/assignment_loss‚3.688399969585358 /loss/FRt3/detection_loss‚2.1740270337710776 1loss/SRqqt1/detection_loss‚2.7167458732922873 0loss/FRt3/assignment_loss‚2.9506035844484964 2loss/SRqqt2/assignment_loss‚1.9693117502440767 0loss/FBt2/assignment_loss‚0.6248163506388664 1loss/FBt3/assignment_loss‚0.16094709119897743 1loss/SRqqt2/detection_loss‚1.6148826146452893 0loss/SRqqt4/detection_loss‚2.469173076748848 %loss/total_loss‚57.52888364124426 0loss/FRt1/assignment_loss‚4.6073902816307255 _runtime‚1550.656578101 /loss/FRt4/assignment_loss‚4.238635415616243 0loss/FBt1/assignment_loss‚0.2587110964620409 /loss/FRt2/detection_loss‚2.2348051198891232 epoch‚1 $ _timestamp‚1.788408940244511e+09 1loss/SRqqt1/assignment_loss‚2.507234118201516 0loss/FBt4/assignment_loss‚0.3698678988548636 /loss/FBt1/detection_loss‚3.6642318047009983 1loss/SRqqt3/assignment_loss‚2.148761577497829 .loss/FRt1/detection_loss‚2.765181270314426 .loss/FBt4/detection_loss‚5.436102338812568 trainer/global_step‚2149Û’L1[ÏÕ .loss/FBt4/detection_loss‚5.436102338812568 trainer/global_step‚2149 1loss/SRqqt4/assignment_loss‚2.217841077595949 /loss/FRt4/detection_loss‚3.0310828701957413 1loss/SRqqt3/detection_loss‚1.9745993782173503 /loss/FBt2/detection_loss‚2.3525409251700475 .loss/FBt3/detection_loss‚1.352991123511562 _step‚174 /loss/FRt2/assignment_loss‚3.688399969585358 /loss/FRt3/detection_loss‚2.1740270337710776 1loss/SRqqt1/detection_loss‚2.7167458732922873 0loss/FRt3/assignment_loss‚2.9506035844484964 2loss/SRqqt2/assignment_loss‚1.9693117502440767 0loss/FBt2/assignment_loss‚0.6248163506388664 1loss/FBt3/assignment_loss‚0.16094709119897743 1loss/SRqqt2/detection_loss‚1.6148826146452893 0loss/SRqqt4/detection_loss‚2.469173076748848 %loss/total_loss‚57.52888364124426 0loss/FRt1/assignment_loss‚4.6073902816307255 _runtime‚1550.656578101 /loss/FRt4/assignment_loss‚4.238635415616243 0loss/FBt1/assignment_loss‚0.2587110964620409 /loss/FRt2/detection_loss‚2.2348051198891232 epoch‚1 $ _timestamp‚1.788408940244511e+09 1loss/SRqqt1/assignment_loss‚2.507234118201516 0loss/FBt4/assignment_loss‚0.3698678988548636 /loss/FBt1/detection_loss‚3.6642318047009983 1loss/SRqqt3/assignment_loss‚2.148761577497829 .loss/FRt1/detection_loss‚2.765181270314426®ûž%כЂ(: õèãÔ”Š£™ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.79' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.758000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ‰àµÑ‚(:‰ õèãÔ÷šÏœ disk.sda.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚447.0625* proc.memory.percent‚17.577075958251953 network.recv‚617921 disk.nvme0c0n1.in‚447.0625 disk.sda3.out‚ 69.03515625 disk.sda1.in‚0% disk.nvme0n1.out‚6808.04638671875 disk.rbd2.in‚0 disk.rbd2.out‚ 36.365234375 disk.rbd3.in‚0 network.sent‚ 3.858852e+06 disk.drbd1777.in‚0 disk.nvme0n1.in‚447.0625 disk.rbd3.out‚0$ memory_percent‚99.96030330657959! proc.memory.rssMB‚ 2879.828125 disk.drbd1777.out‚0& proc.memory.availableMB‚ 6.50390625 disk.dm-1.out‚ 69.03515625 disk.rbd0.out‚0' disk.nvme0c0n1.out‚6808.04638671875 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda.out‚ 69.03515625 disk.rbd1.out‚ 0.982421875) disk./.usagePercent‚70.94723202401113 disk.sda2.out‚0" disk.dm-0.out‚6808.04638671875 disk.rbd0.in‚0% disk./.usageGB‚1057.1650276184082 cpu‚25.25689328649043+é˜òÒ‚(:… „éãÔÔ gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.889' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ×W„Ó‚(:„ „éãÔû¯´› disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 69.609375 disk.sda.out‚ 69.609375 disk.dm-1.in‚0$ memory_percent‚99.95787143707275 network.recv‚622997 disk.dm-0.in‚447.0625" disk.dm-0.out‚6907.94677734375 disk.rbd3.in‚0* proc.memory.percent‚17.577075958251953% disk.nvme0n1.out‚6907.94677734375 disk.rbd0.in‚0 disk.rbd3.out‚0 cpu‚25.254993305900125 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 0.982421875 disk.sda2.out‚0! proc.memory.rssMB‚ 2879.828125& proc.memory.availableMB‚ 6.90234375% disk./.usageGB‚1057.1650276184082 disk.drbd1777.in‚0 disk.rbd2.in‚0 proc.cpu.threads‚17 network.sent‚ 3.965458e+06) disk./.usagePercent‚70.94723202401113 disk.rbd1.in‚0 disk.dm-1.out‚ 69.609375 disk.rbd2.out‚ 37.955078125 disk.nvme0c0n1.in‚447.0625 disk.sda2.in‚0' disk.nvme0c0n1.out‚6907.94677734375 disk.nvme0n1.in‚447.0625 disk.sda.in‚0‹lÛ2Ô- _wandbruntime‚1585  _runtime‚1585£»hÅÕ¿ % lr-AdamW/pg2‚9.89851025242998e-05 trainer/global_step‚2199 % _timestamp‚1.7884089751383958e+09 _runtime‚1585.550500867 _step‚175 % lr-AdamW/pg1‚9.89851025242998e-05f¯xöÊÖÄ _runtime‚1585.550500867 _step‚175 % lr-AdamW/pg1‚9.89851025242998e-05 % lr-AdamW/pg2‚9.89851025242998e-05 trainer/global_step‚2199 % _timestamp‚1.7884089751383958e+09¯äüõ‘2×- _wandbruntime‚1585  _runtime‚1585øp Æ'ØÀO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72693508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44140049 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214779904 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7884089751391044e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3086815982592 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3140803520000 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3077941816687 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚61193754328064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64854699823104 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28533423 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚58212675984384 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2199 _runtime‚ 1585.55146284 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28533423 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚64280570310656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44160085 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚58212658945024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16647104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚61193754328064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚64854723763200 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚537 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _step‚176 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚64280570310656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12374377 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3141018299904 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3078106938083 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44140049 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚61193772331008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29020944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16647106 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238720000 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67995503343104 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44160085 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3086984981504 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚64280757312512? 6•Ë'ÙÅO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28533423 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚64280570310656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44160085 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚58212658945024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16647104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚61193754328064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚64854723763200 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚537 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _step‚176 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚64280570310656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12374377 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3141018299904 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3078106938083 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44140049 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚61193772331008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29020944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16647106 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238720000 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67995503343104 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44160085 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3086984981504 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚64280757312512 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72693508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44140049 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214779904 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7884089751391044e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3086815982592 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3140803520000 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3077941816687 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚61193754328064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64854699823104 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28533423 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚58212675984384 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2199 _runtime‚ 1585.55146284 I ÛÊM [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚68026069528064 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚61221023692800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28546082 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3142432816640 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16654483 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚61221041695744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3088209120768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29034252 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚64309419828736 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44179898 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16654481 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚61318115269555 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 J;DeviceStatsMonitor.on_train_batch_end/acta”³"’ive_bytes.all.peak‚ 4912019968 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3088209120768 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚61317933108591 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236609536 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64883636711424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3142220147200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚64309419828736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 % _timestamp‚1.7884089758020618e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚58238602127360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72725980 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44159836 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29033728 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72725980 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58238619166720 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3079330981231 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64883612771328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72705916 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72705916 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212669440 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚61221041695744 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44159836 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12379247éÕ&ÜÏM UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3088378132992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28546082 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚64309419828736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 % _timestamp‚1.7884089758020618e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚58238602127360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72725980 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44159836 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29033728 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72725980 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58238619166720 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3079330981231 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64883612771328 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72705916 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72705916 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212669440 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚61221041695744 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44159836 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12379247 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚68026069528064 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚61221023692800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28546082 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3142432816640 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16654483 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚61221041695744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3088209120768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29034252 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚64309419828736 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44179898 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16654481 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚61318115269555 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3088209120768 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚61317933108591 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236609536 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64883636711424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3142220147200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3156487305216 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚62557240567808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚66300347234304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3211687415808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3147578756491 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45152787 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221071360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3147413635095 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚62557222564864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3156487305216 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚62657413857675 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚2249 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚62557240567808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚62657231696919 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17015954 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3156656304128 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚65713896871936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3156656304128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3211908487168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29662015 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29166371 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 % _timestamp‚1.7884090102726653e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 _runtime‚1620.685039894 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚66300371174400 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65713709870080 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚62557222564864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚65713709870080/³uÌ'ëÆO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3156487305216 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚62657413857675 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚2249 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚62557240567808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚62657231696919 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17015954 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3156656304128 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚65713896871936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3156656304128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3211908487168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29662015 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29166371 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 % _timestamp‚1.7884090102726653e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 _runtime‚1620.685039894 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚66300371174400 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65713709870080 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚62557222564864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚65713709870080 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3156487305216 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚62557240567808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚66300347234304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3211687415808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3147578756491 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45152787 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221071360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3147413635095 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚62557222564864´X]A2ì- _wandbruntime‚1621  _runtime‚1621 î/¬Ñ&íËM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3157880443392 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚66329260182528 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29674159 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚65742372372992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3213315231232 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3148802799639 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45172600 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚69542599353856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚59535778283520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3148967921243 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3158049455616 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3157880443392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29179032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 % _timestamp‚1.7884090109494374e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29179030 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚507 trainer/global_step‚2249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29674668 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74331568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚509 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚62684746204763 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚59535761244160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚62584509932544 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74351632 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74331568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17023333 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204280832 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚62584491929600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚66329284122624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 _step‚181 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚69542371132928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _runtime‚1621.362384594 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74351632 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12650828 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228220928 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚65742559388160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29179030 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17023331 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚65742559388160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29179032 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45172600 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚62584509932544Á´ëÖ&îÐM VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚65742559388160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29179030 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17023331 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚65742559388160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29179032 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45172600 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚62584509932544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3157880443392 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚66329260182528 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29674159 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚65742372372992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3213315231232 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3148802799639 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45172600 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚69542599353856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚59535778283520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3148967921243 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3158049455616 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3157880443392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29179032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 % _timestamp‚1.7884090109494374e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29179030 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚507 trainer/global_step‚2249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29674668 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74331568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚509 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚62684746204763 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚59535761244160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚62584509932544 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74351632 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74331568 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17023333 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204280832 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚62584491929600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚66329284122624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 _step‚181 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚69542371132928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _runtime‚1621.362384594 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74351632 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12650828 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228220928µä/Ä 2ï- _wandbruntime‚1621  _runtime‚1621¤ûópðÑ 0loss/FRt4/assignment_loss‚4.3082205251507135 1loss/SRqqt1/detection_loss‚2.6576202789942425 .loss/FBt4/detection_loss‚3.417850041673297 _step‚182 % _timestamp‚1.7884090109517925e+09 0loss/FBt4/assignment_loss‚0.2838167674230395 0loss/FRt3/assignment_loss‚3.7220725059509276 0loss/FBt3/assignment_loss‚0.3334760731301079 /loss/FRt3/detection_loss‚2.3036094099283217 0loss/SRqqt4/detection_loss‚2.010150940394869 trainer/global_step‚2249 0loss/FRt1/assignment_loss‚4.5217269884573446 /loss/FRt1/detection_loss‚2.7602389744810156 1loss/SRqqt2/detection_loss‚1.5237538093701004 .loss/FBt1/detection_loss‚3.727957692146301 &loss/total_loss‚57.305534596448396 _runtime‚1621.363829086 2loss/SRqqt2/assignment_loss‚1.8169860012829304 0loss/FBt2/assignment_loss‚0.8386078208238662 /loss/FBt3/detectionÑmú‹ç_loss‚1.0814137161616557 .loss/FRt2/detection_loss‚2.332967654934951 .loss/FRt4/detection_loss‚2.259738800318345 .loss/FBt2/detection_loss‚4.214238675435408 epoch‚1 .loss/FRt2/assignment_loss‚4.01869871360915 2loss/SRqqt1/assignment_loss‚2.8745634237925213 0loss/SRqqt3/assignment_loss‚2.38196244391433 1loss/SRqqt3/detection_loss‚1.7488791793584824 2loss/SRqqt4/assignment_loss‚1.7717197254007937 1loss/FBt1/assignment_loss‚0.39526443431568625åXy\ñÖ 0loss/FBt4/assignment_loss‚0.2838167674230395 0loss/FRt3/assignment_loss‚3.7220725059509276 0loss/FBt3/assignment_loss‚0.3334760731301079 /loss/FRt3/detection_loss‚2.3036094099283217 0loss/SRqqt4/detection_loss‚2.010150940394869 trainer/global_step‚2249 0loss/FRt1/assignment_loss‚4.5217269884573446 /loss/FRt1/detection_loss‚2.7602389744810156 1loss/SRqqt2/detection_loss‚1.5237538093701004 .loss/FBt1/detection_loss‚3.727957692146301 &loss/total_loss‚57.305534596448396 _runtime‚1621.363829086 2loss/SRqqt2/assignment_loss‚1.8169860012829304 0loss/FBt2/assignment_loss‚0.8386078208238662 /loss/FBt3/detection_loss‚1.0814137161616557 .loss/FRt2/detection_loss‚2.332967654934951 .loss/FRt4/detection_loss‚2.259738800318345 .loss/FBt2/detection_loss‚4.214238675435408 epoch‚1 .loss/FRt2/assignment_loss‚4.01869871360915 2loss/SRqqt1/assignment_loss‚2.8745634237925213 0loss/SRqqt3/assignment_loss‚2.38196244391433 1loss/SRqqt3/detection_loss‚1.7488791793584824 2loss/SRqqt4/assignment_loss‚1.7717197254007937 1loss/FBt1/assignment_loss‚0.39526443431568625 0loss/FRt4/assignment_loss‚4.3082205251507135 1loss/SRqqt1/detection_loss‚2.6576202789942425 .loss/FBt4/detection_loss‚3.417850041673297 _step‚182 % _timestamp‚1.7884090109517925e+09¶ 8¸ò‚(:… ÀéãÔɣϙ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.515' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.703 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”/†ó‚(:† ÀéãÔ¼Œì› disk.sda1.out‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 451.03515625 disk.rbd2.in‚0 disk.rbd3.out‚0! proc.memory.rssMB‚ 2879.828125! disk.nvme0c0n1.out‚ 7206.40625) disk./.usagePercent‚70.94725250461387 disk.drbd1777.in‚0" disk.nvme0c0n1.in‚ 451.03515625& proc.memory.availableMB‚ 6.80078125 disk.nvme0n1.in‚ 451.03515625 disk.rbd2.out‚38.22119140625 disk.sda2.in‚0$ memory_percent‚99.95849132537842* proc.memory.percent‚17.577075958251953 disk.nvme0n1.out‚ 7206.40625 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 72.05078125 disk.dm-1.in‚0 disk.sda.out‚ 72.05078125 disk.dm-0.out‚ 7206.40625 network.recv‚641858 disk.sda1.in‚0 disk.rbd1.out‚ 0.982421875 disk.dm-1.out‚ 72.05078125 cpu‚25.247417430352836 disk.rbd3.in‚0 disk.sda3.in‚0 disk.sda.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1057.1653327941895 disk.rbd1.in‚0 disk.rbd0.out‚0 network.sent‚ 4.090917e+06åÆ…œô‚(:‘ ÏéãÔ€ÀŸ™ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚110.267' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.053399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óDÕ¢õ‚(:‹ ÏéãÔ÷踜 disk.dm-1.in‚0% disk./.usageGB‚1057.1653137207031 disk.sda.out‚ 72.73828125 disk.rbd1.in‚0 disk.dm-0.out‚ 7206.6640625 disk.rbd0.out‚0 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 451.14453125 disk.sda2.out‚0! proc.memory.rssMB‚ 2879.828125 proc.cpu.threads‚17 disk.dm-0.in‚ 451.14453125 disk.dm-1.out‚ 72.73828125 disk.rbd2.in‚0 network.sent‚ 4.198842e+06 disk.drbd1777.out‚0 disk.rbd2.out‚38.42431640625 disk.sda3.in‚0 disk.rbd3.in‚0 network.recv‚647803 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 451.14453125 disk.rbd0.in‚0& proc.memory.availableMB‚ 6.54296875* proc.memory.percent‚17.577075958251953 cpu‚25.245780032523495 disk.sda3.out‚ 72.73828125 disk.sda1.in‚0# disk.nvme0c0n1.out‚ 7206.6640625 disk.sda2.in‚0$ memory_percent‚99.96006488800049( disk./.usagePercent‚70.9472512245762 disk.sda.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 0.982421875! disk.nvme0n1.out‚ 7206.6640625;’Bä2ö- _wandbruntime‚1656  _runtime‚1656ÍyøSÇ÷Á _runtime‚1656.074528821 _step‚183 & lr-AdamW/pg1‚9.855370731957889e-05 & lr-AdamW/pg2‚9.855370731957889e-05 trainer/global_step‚2299 % _timestamp‚1.7884090456623905e+09àã9œÌøÆ % _timestamp‚1.7884090456623905e+09 _runtime‚1656.074528821 _step‚183 & lr-AdamW/pg1‚9.855370731957889e-05 & lr-AdamW/pg2‚9.855370731957889e-05 trainer/global_step‚2299·Þ>Ï_2ù- _wandbruntime‚1656  _runtime‚1656¡‡³RÇ'úÁO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64023862632127 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46145489 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚60806977178624 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30302197 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚1656.075893702 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3226158627840 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3226327626752 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3282821386752 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682752 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚67147036431360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚63920708804608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3226158627840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29799323 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75924774 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚67147036431360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30302708 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3282608704000 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚63920690801664 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚67146849429504 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚511 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67746018585600 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3216885453503 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75944812 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12917902 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71028603349504 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46145489 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71028839972352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3217050574899 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17384806 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29799323 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚60806977178624 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30302197 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚1656.075893702 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3226158627840 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3226327626752 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3282821386752 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682752 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚67147036431360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚63920708804608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3226158627840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29799323 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75924774 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚67147036431360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30302708 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3282608704000 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚63920690801664 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚67146849429504 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚511 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67746018585600 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3216885453503 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75944812 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12917902 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71028603349504 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46145489 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71028839972352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3217050574899 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17384806 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29799323 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12922658 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚67175511932416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17392183 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71058938825728 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67774907593728 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3218274618047 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚63947978169344 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚63947978169344 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46145240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75977284 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _step‚185 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3218439739651 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29811982 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46165302 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46165302 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30315335 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30314839 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12923152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202183680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3227551766016 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3227551766016 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75977284 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚494 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17392181 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3284031232000 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 % _timestamp‚1.7884090463076582e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64051377139971 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71059164949504 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚67175698947584 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3227720778240 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚63947960166400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/o©Ý{…… versize_segments.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30314839 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12923152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202183680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3227551766016 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3227551766016 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75977284 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚494 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17392181 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3284031232000 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 % _timestamp‚1.7884090463076582e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64051377139971 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71059164949504 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚67175698947584 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3227720778240 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚63947960166400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12922658 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚67175511932416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17392183 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71058938825728 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248¹6ÐýÊ2ÿ- _wandbruntime‚1656  _runtime‚1656‰ ±U€Ï 0loss/SRqqt1/assignment_loss‚1.91981141269207 2loss/SRqqt3/assignment_loss‚2.2454920031276404 /loss/FBt4/detection_loss‚4.0595502853393555 %loss/total_loss‚58.29867769703382 0loss/FRt3/assignment_loss‚3.7892607549826303 /loss/FRt4/assignment_loss‚4.852891535986037 /loss/FRt3/detection_loss‚2.0272690590333053 0loss/SRqqt2/detection_loss‚1.643014489820129 0loss/SRqqt4/detection_loss‚2.189550185865826 epoch‚1 % _timestamp‚1.7884090463094292e+09 2loss/SRqqt2/assignment_loss‚1.9285547401952117 0loss/FBt1/assignment_loss‚0.3795050404822372 trainer/global_step‚2299 0loss/FBt4/assignment_loss‚0.6192096510615427 /loss/FBt2/detection_loss‚2.6757512185384256 _runtime‚1656.721815001 0loss/FRt2/assignment_loss‚3.9109120605126866 2loss/SRqqt4/assignment_loss‚1.9772189146942563 /loss/FRt4/detection_loss‚2.2316383648486364 0loss/SRqqt1/detection_loss‚3.543539398908615 .loss/FBt3/detection_loss‚1.763941551651867 _step‚186 /loss/FRt1/detection_loss‚3.0870825797319412 /loss/FRt1/assignment_loss‚4.905915684998035 0loss/FBt3/assignment_loss‚0.3008304771645271 0loss/SRqqt3/detection_loss‚1.948938708533259 0loss/FBt2/assignment_loss‚0.5688989281654357 /loss/FRt2/detection_loss‚2.4915956759227895 .loss/FBt1/detection_loss‚3.238304974777358Msx—ZÔ _runtime‚1656.721815001 0loss/FRt2/assignment_loss‚3.9109120605126866 2loss/SRqqt4/assignment_loss‚1.9772189146942563 /loss/FRt4/detection_loss‚2.2316383648486364 0loss/SRqqt1/detection_loss‚3.543539398908615 .loss/FBt3/detection_loss‚1.763941551651867 _step‚186 /loss/FRt1/detection_loss‚3.0870825797319412 /loss/FRt1/assignment_loss‚4.905915684998035 0loss/FBt3/assignment_loss‚0.3008304771645271 0loss/SRqqt3/detection_loss‚1.948938708533259 0loss/FBt2/assignment_loss‚0.5688989281654357 /loss/FRt2/detection_loss‚2.4915956759227895 .loss/FBt1/detection_loss‚3.238304974777358 0loss/SRqqt1/assignment_loss‚1.91981141269207 2loss/SRqqt3/assignment_loss‚2.2454920031276404 /loss/FBt4/detection_loss‚4.0595502853393555 %loss/total_loss‚58.29867769703382 0loss/FRt3/assignment_loss‚3.7892607549826303 /loss/FRt4/assignment_loss‚4.852891535986037 /loss/FRt3/detection_loss‚2.0272690590333053 0loss/SRqqt2/detection_loss‚1.643014489820129 0loss/SRqqt4/detection_loss‚2.189550185865826 epoch‚1 % _timestamp‚1.7884090463094292e+09 2loss/SRqqt2/assignment_loss‚1.9285547401952117 0loss/FBt1/assignment_loss‚0.3795050404822372 trainer/global_step‚2299 0loss/FBt4/assignment_loss‚0.6192096510615427 /loss/FBt2/detection_loss‚2.6757512185384256º\—B‘‚‚(:† ÞéãÔ±ËÆ™ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.917' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3834 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úÛ~ƒ‚(:“ ÞéãÔýž­œ network.sent‚ 4.205034e+06 disk.rbd1.in‚0! proc.memory.rssMB‚ 2879.828125 disk.rbd2.in‚0 disk.rbd2.out‚39.95166015625 disk.sda3.out‚ 73.14453125 disk.sda.out‚ 73.14453125 disk.sda1.out‚0 disk.nvme0n1.in‚ 451.14453125 network.recv‚652641 disk.dm-1.out‚ 73.14453125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 451.14453125! disk.dm-0.out‚7305.9580078125& proc.memory.availableMB‚ 6.75390625" disk.nvme0c0n1.in‚ 451.14453125$ disk./.usageGB‚1057.165370941162 disk.rbd3.out‚0 disk.drbd1777.out‚0$ memory_percent‚99.95877742767334* proc.memory.percent‚17.577075958251953 disk.sda.in‚0 disk.rbd1.out‚ 0.982421875$ disk.nvme0n1.out‚7305.9580078125 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda1.in‚0( disk./.usagePercent‚70.9472550646892 disk.dm-1.in‚0& disk.nvme0c0n1.out‚7305.9580078125 cpu‚25.243896436184997(|¯Ê„‚(:… íéãÔŸë²™ gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚88.889' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËT+9"…‚(:— íéãÔÓò¼œ' disk.nvme0c0n1.out‚7307.26611328125 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 73.72265625 disk.sda1.in‚0 disk.sda3.out‚ 73.72265625 disk.sda.in‚0 disk.nvme0n1.in‚ 451.14453125 disk.dm-1.in‚0 disk.dm-1.out‚ 73.72265625% proc.memory.availableMB‚ 6.1953125 cpu‚25.242039820600404 network.recv‚659199 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd1.out‚ 0.982421875 disk.sda2.in‚0 disk.rbd3.out‚0 disk.drbd1777.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚40.07666015625 disk.sda2.out‚0 disk.drbd1777.out‚0" disk.dm-0.out‚7307.26611328125$ memory_percent‚99.96218681335449 network.sent‚ 4.313344e+06) disk./.usagePercent‚70.94601394016321% disk./.usageGB‚1057.1468772888184 disk.rbd2.in‚0! proc.memory.rssMB‚ 2879.828125* proc.memory.percent‚17.577075958251953" disk.nvme0c0n1.in‚ 451.14453125 disk.sda3.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚7307.26611328125 disk.dm-0.in‚ 451.14453125ÌÝ–2†- _wandbruntime‚1691  _runtime‚1691Š¡iLJÁ trainer/global_step‚2349 % _timestamp‚1.7884090808632631e+09 _runtime‚1691.275444609 _step‚187 & lr-AdamW/pg1‚9.832297067217974e-05 & lr-AdamW/pg2‚9.832297067217974e-05M¹ÌÌˆÆ % _timestamp‚1.7884090808632631e+09 _runtime‚1691.275444609 _step‚187 & lr-AdamW/pg1‚9.832297067217974e-05 & lr-AdamW/pg2‚9.832297067217974e-05 trainer/global_step‚2349»)q›-2‰- _wandbruntime‚1691  _runtime‚1691%1Ç'ŠÁO TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚65390493567335 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚72545223680512 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚69191642056704 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3295829950464 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _step‚188 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30432273 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚65284159038464 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30432273 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚65284177041408 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚65390675728091 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13189763 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622848 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68579988988928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3353581623808 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚68580175990784 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚65284159038464 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30943983 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚69191665996800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚2349 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3295998949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚72545460303360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚62104153334784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47138191 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3286522393307 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3286357271911 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _runtime‚1691.276436969 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47118155 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30432271 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚564 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚68579988988928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77570464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0îìì„< ‹ÆO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚65284177041408 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚65390675728091 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13189763 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytÆÓes.all.current‚ 236622848 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68579988988928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3353581623808 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚68580175990784 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚65284159038464 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30943983 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚69191665996800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚2349 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3295998949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚72545460303360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚62104153334784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47138191 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3286522393307 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3286357271911 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _runtime‚1691.276436969 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47118155 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30432271 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚564 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚68579988988928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77570464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚65390493567335 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚72545223680512 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚69191642056704 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3295829950464 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _step‚188 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30432273 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚65284159038464 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30432273 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0¼µè0ò2Œ- _wandbruntime‚1691  _runtime‚1691gr<Ñ&ËM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚65311428403200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3297223088640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17761031 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13195133 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2349 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30956164 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17761033 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3355002591232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47137942 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30444932 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77582872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 % _timestamp‚1.7884090815343516e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47158004 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚65311428403200 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚546 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77582872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68608651491840 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚72575557596160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77602936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47158004 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3297223088640 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚68608838507008 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚72575790011392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3287911558059 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3287746436455 _runtime‚1691.947357826 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚72575557596160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77602936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47158004 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3297223088640 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚68608838507008 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚72575790011392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3287911558059 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3287746436455 _runtime‚1691.947357826 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚65311428403200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3297223088640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17761031 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13195133 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2349 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30956164 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17761033 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3355002591232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47137942 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30444932 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77582872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 % _timestamp‚1.7884090815343516e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47158004 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚65311428403200 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚546 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77582872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68608651491840½ÁŸ©¸2- _wandbruntime‚1691  _runtime‚1691׃¾/RÌ 0loss/FBt1/assignment_loss‚0.3045850134363718 /loss/FRt3/detection_loss‚2.2644015691346593 # _timestamp‚1.78840908153656e+09 .loss/FBt1/detection_loss‚5.412494831283888 epoch‚1 trainer/global_step‚2349 0loss/FBt2/assignment_loss‚0.7660731431096792 0loss/FBt3/assignment_loss‚0.3773987165311403 .loss/FRt2/detection_loss‚2.117469950289023 /loss/FRt4/detection_loss‚2.2879137049118676 0loss/FRt3/assignment_loss‚2.7782485405604045 0loss/SRqqt1/detection_loss‚2.869280910917691 2loss/SRqqt1/assignment_loss‚3.0197246032101766 2loss/SRqqt4/assignment_loss‚1.8350495225658603 1loss/SRqqt4/detection_loss‚1.8684938363000458 /loss/FBt2/detection_loss‚3.5755773562788287 _runtime‚1691.948671781 .loss/FRt1/assignment_loss‚4.75905327796936 1loss/SRqqt3/assignment_loss‚1.801715521674071 0loss/SRqqt3/detection_loss‚1.827800822311214 /loss/FRt4/assignment_loss‚4.464804224669933 2loss/SRqqt2/assignment_loss‚1.5418600832302887 .loss/FBt4/detection_loss‚3.395108211785555 %loss/total_loss‚57.62910153840815 _step‚190 /loss/FRt2/assignment_loss‚4.039572387445168 /loss/FBt4/assignment_loss‚0.525714337847603 /loss/FRt1/detection_loss‚2.8138530237334116 1loss/SRqqt2/detection_loss‚1.5761308686201831 /loss/FBt3/detection_loss‚1.4067770805917297x¯W‘Ñ 0loss/FBt1/assignment_loss‚0.3045850134363718 /loss/FRt3/detection_loss‚2.2644015691346593 # _timestamp‚1.78840908153656e+09 .loss/FBt1/detection_loss‚5.412494831283888 epoch‚1 trainer/global_step‚2349 0loss/FBt2/assignment_loss‚0.7660731431096792 0loss/FBt3/assignment_loss‚0.3773987165311403 .loss/FRt2/detection_loss‚2.117469950289023 /loss/FRt4/detection_loss‚2.2879137049118676 0loss/FRt3/assignment_loss‚2.7782485405604045 0loss/SRqqt1/detection_loss‚2.869280910917691 2loss/SRqqt1/assignment_loss‚3.0197246032101766 2loss/SRqqt4/assignment_loss‚1.8350495225658603 1loss/SRqqt4/detection_loss‚1.8684938363000458 /loss/FBt2/detection_loss‚3.5755773562788287 _runtime‚1691.948671781 .loss/FRt1/assignment_loss‚4.75905327796936 1loss/SRqqt3/assignment_loss‚1.801715521674071 0loss/SRqqt3/detection_loss‚1.827800822311214 /loss/FRt4/assignment_loss‚4.464804224669933 2loss/SRqqt2/assignment_loss‚1.5418600832302887 .loss/FBt4/detection_loss‚3.395108211785555 %loss/total_loss‚57.62910153840815 _step‚190 /loss/FRt2/assignment_loss‚4.039572387445168 /loss/FBt4/assignment_loss‚0.525714337847603 /loss/FRt1/detection_loss‚2.8138530237334116 1loss/SRqqt2/detection_loss‚1.5761308686201831 /loss/FBt3/detection_loss‚1.4067770805917297¾•·A’‚(:… üéãÔû®™™ gpu.0.gpu‚20.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚90.786' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.1572 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 £Q “‚(:€ üéãԜྛ disk.nvme0c0n1.in‚ 453.296875" disk.nvme0c0n1.out‚ 7406.390625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 453.296875 disk.dm-0.out‚ 7406.390625 disk.rbd0.in‚0 disk.sda3.out‚ 74.0859375 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd3.in‚0& proc.memory.availableMB‚ 6.95703125 network.recv‚662922 disk.sda.out‚ 74.0859375 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0* proc.memory.percent‚17.577075958251953 network.sent‚ 4.318752e+06) disk./.usagePercent‚70.94607871006939 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 0.982421875 disk.nvme0n1.in‚ 453.296875 disk.nvme0n1.out‚ 7406.390625 disk.rbd2.out‚40.09228515625% disk./.usageGB‚1057.1478424072266 disk.dm-1.in‚0 cpu‚25.240249406011568 disk.drbd1777.out‚0 disk.dm-1.out‚ 74.0859375$ memory_percent‚99.95753765106201! proc.memory.rssMB‚ 2879.828125x7"œ”‚(:‘ ‹êãÔõ¶™ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.341' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.868199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·#/Œ•‚(:„ ‹êãÔ¢º‰œ disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.drbd1777.in‚0! proc.memory.rssMB‚ 2879.828125 disk.rbd2.in‚0 disk.rbd2.out‚ 40.2451171875 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 0.982421875$ memory_percent‚99.95765686035156) disk./.usagePercent‚70.94607358991868 disk.sda1.out‚0 disk.dm-0.in‚ 453.296875 proc.cpu.threads‚17& disk.nvme0c0n1.out‚7505.6669921875 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 453.296875 disk.rbd1.in‚0 disk.nvme0n1.in‚ 453.296875$ disk.nvme0n1.out‚7505.6669921875 network.recv‚668644 disk.dm-1.in‚0 disk.sda3.out‚ 74.734375" proc.memory.availableMB‚6.9375 network.sent‚ 4.426559e+06 disk.dm-1.out‚ 74.734375 disk.sda1.in‚0% disk./.usageGB‚1057.1477661132812 disk.sda3.in‚0 disk.sda.out‚ 74.734375* proc.memory.percent‚17.577075958251953 cpu‚25.238586463450236 disk.rbd0.out‚0 disk.drbd1777.out‚0! disk.dm-0.out‚7505.6669921875L)’"–‚(:… šêãÔð«š gpu.0.gpu‚44.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.353' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0706 gpu.0.smClæ=cônock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šåF—‚(:„ šêãÔá‘Ôœ disk.nvme0n1.in‚ 460.921875 disk.sda2.in‚0 disk.rbd1.in‚0! proc.memory.rssMB‚ 2879.828125 disk.sda1.in‚0" disk.nvme0n1.out‚ 7506.39453125 disk.dm-1.in‚0$ memory_percent‚99.95770454406738 disk.sda3.in‚0 disk.sda.out‚ 75.08984375 disk.dm-1.out‚ 75.08984375 disk.rbd2.out‚ 40.2607421875 proc.cpu.threads‚17( disk./.usagePercent‚70.9460751259639 disk.sda3.out‚ 75.08984375 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 0.982421875% proc.memory.availableMB‚ 6.9296875% disk./.usageGB‚1057.1477890014648 disk.sda2.out‚0 disk.dm-0.out‚ 7506.39453125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 cpu‚25.2369526478287 network.recv‚673723 disk.sda.in‚0 disk.dm-0.in‚ 460.921875 disk.rbd2.in‚0 disk.rbd3.in‚0* proc.memory.percent‚17.577075958251953 network.sent‚ 4.432697e+06 disk.nvme0c0n1.in‚ 460.921875$ disk.nvme0c0n1.out‚ 7506.39453125Þí®î2˜- _wandbruntime‚1726  _runtime‚1726õ \¿Ç™Á & lr-AdamW/pg1‚9.808227141291204e-05 & lr-AdamW/pg2‚9.808227141291204e-05 trainer/global_step‚2399 % _timestamp‚1.7884091159193485e+09 _runtime‚1726.331514946 _step‚191œ¡ŸŒÌšÆ & lr-AdamW/pg1‚9.808227141291204e-05 & lr-AdamW/pg2‚9.808227141291204e-05 trainer/global_step‚2399 % _timestamp‚1.7884091159193485e+09 _runtime‚1726.331514946 _step‚191¿ªš7¤2›- _wandbruntime‚1726  _runtime‚1726 ž™±Ç'œÁO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238720000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31065223 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48130893 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3424696081408 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 _step‚192 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚70013315550208 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3365501273088 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚70013315550208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70013128548352 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚70013128548352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13463077 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚66647627275264 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18122506 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214779904 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚66757306663299 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48110857 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚74061770769408 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚63401295412224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚66647627275264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18122504 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚70637313408000 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚1726.332621832 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31065223 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63401312451584 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48130893 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48110857 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚2399 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79196116 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚70637289467904 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70013128548352 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚70013128548352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13463077 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚66647627275264 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18122506 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214779904 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚66757306663299 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48110857 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚74061770769408 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚63401295412224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚66647627275264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18122504 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚70637313408000 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚1726.332621832 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31065223 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63401312451584 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48130893 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48110857 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚2399 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79196116 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚70637289467904 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48150706 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63427255633920 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2399 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13469313 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚70666202416128 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66784639010387 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66784456849423 _step‚193 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚66674896640000 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79228588 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79208524 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3357218254863 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63427238594560 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3366894411264 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79208524 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214766592 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _runtime‚ 1726.98982012 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31599196 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31077880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18129883 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3425897796096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3367063423488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18129881 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3357383376467 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 % _timestamp‚1.7884091165771306e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31598658 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚66674914642944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚66674914642944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31077882 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31077880 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48130644 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚66674896640000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3367063423488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3426112562688 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74092100212224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3366894411264 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968CFN²ª ÏM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74092338918912 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238706688 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31077882 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70041978066432 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚70041791051264 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48150706 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 SDDeviceStaÖß+$tsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70041978066432 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13468777 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74092100212224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3366894411264 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48150706 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63427255633920 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2399 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13469313 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚70666202416128 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66784639010387 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66784456849423 _step‚193 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚66674896640000 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79228588 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79208524 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3357218254863 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63427238594560 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3366894411264 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79208524 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214766592 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _runtime‚ 1726.98982012 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31599196 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31077880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18129883 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3425897796096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3367063423488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18129881 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3357383376467 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 % _timestamp‚1.7884091165771306e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31598658 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚66674914642944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚66674914642944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31077882 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31077880 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48130644 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚66674896640000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3367063423488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3426112562688 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15975 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24399 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 127307264 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚64771409045504 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 123925584 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 417333248 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 127307264 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚281 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚283 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 265683968 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚72147234137088 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 _step‚195 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚48869120 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 3409137449984 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚80683775 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚32388997 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15977 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚199 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚217 _runtime‚1787.521287598 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 410711040 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚32389280 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚416 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24126 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 133611008 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 4757245249 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚217 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚199 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚199 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24126 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚26 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 109670912 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚68090157312000 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚80683775 VCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚71499276759040 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 4912019968 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 3409137449984 $ _timestamp‚1.788409177108654e+09 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚13807830 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚68170848550799 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 406789961 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚80699752 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚18581167 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚64771392006144 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚80699752 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚68090139309056 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚31830632 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚68090139309056 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚48869120 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚18581169 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚217 YFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚71499276759040 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚13808111 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚416™ë™ ®¬T QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚80683775 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚32388997 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15977 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚199 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚217 _runtime‚1787.521287598 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 410711040 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚32389280 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚416 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24126 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 133611008 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 4757245249 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚217 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚199 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚199 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24126 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚26 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 109670912 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚68090157312000 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚80683775 VCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚71499276759040 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 4912019968 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 3409137449984 $ _timestamp‚1.788409177108654e+09 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚13807830 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚68170848550799 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 406789961 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚80699752 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚18581167 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚64771392006144 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚80699752 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚68090139309056 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚31830632 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚68090139309056 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚48869120 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚18581169 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚217 YFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚71499276759040 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚13808111 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚416 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ZGDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚71499422069248 ^KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚75615903130624 YFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚68170707585855 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 3399315579711 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15975 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24399 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 127307264 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚64771409045504 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 123925584 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 417333248 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 127307264 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚281 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚283 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 265683968 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚72147234137088 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 _step‚195 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚48869120 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 3409137449984ÃÈìC2¯- _wandbruntime‚1788  _runtime‚1788X]Áú8)°²R TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚32400734 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚217 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 4541568000 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2460 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚217 trainer/global_step‚2417 _step‚196 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 4624220160 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24126 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚31841344 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 4912019968 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 3409486585856 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 410711040 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚72166837702656 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 5041553408 TADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚71518491080704 WDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚71518491080704 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1003 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚416 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24126 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 133611008 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24399 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚31841342 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚32400451 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚13813065 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15977 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 127307264 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 3409486585856 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 410711040 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚13812784 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚217 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚31841342 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 265683968 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 3469042069504 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 417333248 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 127307264 `MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚75635989443072 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 3399663186047 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 145310208 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚68109022497792 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚48860541 _runtime‚1788.441313929 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚18587667 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 3399787111631 \IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚75635855832064 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 109670912 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚26 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚80717860 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚31841344s—«”=)±·R GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 140964944 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚416 XEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚71518636390912 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚281 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚64789310599168 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚283 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚18587669 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚68188956745855 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15975 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚72166813762560 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚416 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚64789293559808 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚199 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚68109004494848 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 3409613893120 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 3409613893120 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚199 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚80701883 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 406789961 % _timestamp‚1.7884091780287433e+09 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚982 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 4757245249 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24399 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚31841342 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚32400451 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚13813065 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15977 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 127307264 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 3409486585856 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 410711040 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚13812784 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚217 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚31841342 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 265683968 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 3469042069504 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 417333248 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 127307264 `MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚75635989443072 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 3399663186047 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 145310208 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚68109022497792 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚48860541 _runtime‚1788.441313929 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚18587667 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 3399787111631 \IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚75635855832064 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 109670912 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚26 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚80717860 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚31841344 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚32400734 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚217 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 4541568000 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2460 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚217 trainer/global_step‚2417 _step‚196 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 4624220160 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24126 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚31841344 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 4912019968 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 3409486585856 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 410711040 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚72166837702656 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 5041553408 TADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚71518491080704 WDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚71518491080704 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1003 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚416 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24126 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 133611008 B&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.2081337422132492 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.011904762126505375 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.003500203834846616 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.006235827691853046 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_5‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.16741284728050232 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.21500132977962494 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.21001777052879333 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04399823769927025 particle/accuracy_1_of_1‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.3984259068965912 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13752378523349762 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.36006835103034973 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.2625174820423126 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1882360577583313 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.32499998807907104 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.309027761220932 *jet/accuracy_1_of_6‚0.8910256624221802 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08726852387189865 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.23472224175930023 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 _step‚197 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4962027668952942 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9925066232681274 6 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.111111119389534 ,jet/accuracy_3_of_4‚0.032680198550224304 *jet/accuracy_1_of_5‚0.8166539072990417 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.26524874567985535 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.09996459633111954 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.0833333358168602 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.25567764043807983 6 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.209196075797081 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 particle/accuracy_2_of_5‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 % _timestamp‚1.7884091902333477e+09 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16011983156204224 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.20025849342346191 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900468587875366 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3037280738353729 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.35879629850387573 particle/accuracy_5_of_6‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.10164283215999603 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.550000011920929 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.21169565618038177 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.39201605319976807 particle/accuracy_5_of_5‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3199999928474426 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 trainer/global_step‚2417 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.11108061671257019 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.26212167739868164 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.26870065927505493 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.40408530831336975 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9909008741378784 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.26655325293540955 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.1458333283662796 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.16831354796886444 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2265331894159317 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.15833333134651184 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 6 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.103448286652565 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_7_of_7‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.36934366822242737 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.24051563441753387 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17284747958183289 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.21662481129169464 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.007956549525260925 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.33763739466667175 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.006966649554669857 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3347596824169159 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.10112474858760834 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.022832242771983147 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.29629629850387573 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17480002343654633 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.44028419256210327 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.08160265535116196 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.26584821939468384 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16939088702201843 +jet/accuracy_2_of_2‚0.07921116799116135 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.1849086880683899 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.42752814292907715 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9892454147338867 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1596117913722992 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,jet/accuracy_3_of_5‚0.050358422100543976 particle/accuracy_3_of_3‚1 particle/accuracy_2_of_4‚1 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.1794973611831665 'particle/accuracy‚0.838500440120697 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03144654259085655 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.00382082792930305 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9975579977035522 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.19462814927101135 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.265625 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.20132488012313843 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0038746020291000605 +jet/accuracy_2_of_3‚0.18484266102313995 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.20372425019741058 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.2156156599521637 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2053571492433548 +particle/specificity‚0.9460254311561584 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9978796243667603 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.256729394197464 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.015796637162566185 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.2537330985069275 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.18518516421318054 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.011904762126505375 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6111111044883728 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.24795906245708466 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.29769957065582275 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,particle/sensitivity‚0.25744056701660156 particle/accuracy_3_of_6‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9962522387504578 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.40625 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9882514476776123 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.125 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.30000001192092896 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.1820702701807022 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.07280120253562927 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 7validation_average_jet_accuracy‚0.32777780294418335 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.2458639144897461 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.1928160935640335 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.00692239822819829 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.08484699577093124 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3778073489665985 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_4_of_5‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚56.99307222514535 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5151515007019043 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.16995464265346527 particle/accuracy_1_of_2‚1 jet/accuracy_5_of_6‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1702127605676651 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9298245310783386 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3856906592845917 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.5972180366516113 jet/accuracy_4_of_6‚0 particle/accuracy_6_of_6‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6425621509552002 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9841716885566711 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.18209876120090485 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_6‚0.33974361419677734 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1669503003358841 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.25057318806648254 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.23173238337039948 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3860960900783539 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_4_of_4‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18462078273296356 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9833787083625793 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9647988080978394 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.05367966368794441 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.1666666716337204 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9338061809539795 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9958333373069763 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.24425287544727325 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 epoch‚1 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 6 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.302325576543808 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.13963963091373444 particle/accuracy_2_of_6‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986336827278137 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.17573584616184235 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_7‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.1458333283662796 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9881753921508789 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.45575061440467834 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.007921649143099785 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.24074074625968933 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 _runtime‚1800.648542025 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.7043626308441162 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9747664332389832 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3670634627342224 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.2800000011920929 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9969736933708191 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9925458431243896 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7"Purity/*FRt0SRqqt*FBt/event_purity‚0.33919557929039 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.19342045485973358 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16287237405776978 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1858585923910141 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.15922456979751587 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3415229916572571 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.10299309343099594 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2531985342502594 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15628190338611603 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1826576590538025 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.11919737607240677 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_6‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.21052631735801697 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.1699240356683731 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 *jet/accuracy_1_of_4‚0.7813287377357483 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.2813165485858917 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.27005478739738464 jet/accuracy_7_of_7‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.11245917528867722 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.19364793598651886 particle/accuracy_1_of_3‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19681648910045624 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1376984566450119 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.26657429337501526 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365078806877136 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.006699909456074238 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.1701527237892151 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6341173648834229 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21666666865348816 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.01612045429646969 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.159820556640625 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8256692290306091 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9915631413459778 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2388889044523239 *"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 *"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1597222238779068 particle/accuracy_6_of_7‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722350239753723 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.054946135729551315 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.414144903421402 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9915900230407715 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.37825271487236023 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.024385560303926468 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 particle/accuracy_2_of_2‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.21757657825946808 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.33712121844291687 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.3941907584667206 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1791728138923645 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_3_of_5‚1 jet/accuracy_4_of_7‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3392857015132904 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.17476283013820648 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17667415738105774 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3499338626861572 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9893838167190552 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9851943254470825 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.36769968271255493 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.2078947275876999 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0972222313284874 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.09996459633111954 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.3943452537059784 ("Puri,¹ó†£ty/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 particle/accuracy_1_of_7‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9950373768806458 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.01708628237247467 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 )jet/accuracy_2_of_5‚0.342626690864563 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.1809368133544922 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 jet/accuracy_5_of_5‚0 particle/accuracy_3_of_4‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.18595680594444275 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.2023809552192688 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.14057238399982452 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_6_of_7‚0 particle/accuracy_4_of_7‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.21495021879673004Ò±ÝFOz¶º¤ ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.01612045429646969 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.159820556640625 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8256692290306091 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9915631413459778 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2388889044523239 *"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 *"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1597222238779068 particle/accuracy_6_of_7‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722350239753723 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.054946135729551315 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.414144903421402 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9915900230407715 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.37825271487236023 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.024385560303926468 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 particle/accuracy_2_of_2‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.21757657825946808 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.33712121844291687 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.3941907584667206 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1791728138923645 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_3_of_5‚1 jet/accuracy_4_of_7‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3392857015132904 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.17476283013820648 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17667415738105774 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3499338626861572 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9893838167190552 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9851943254470825 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.36769968271255493 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.2078947275876999 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0972222313284874 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.09996459633111954 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.3943452537059784 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 particle/accuracy_1_of_7‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9950373768806458 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.01708628237247467 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 )jet/accuracy_2_of_5‚0.342626690864563 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.1809368133544922 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 jet/accuracy_5_of_5‚0 particle/accuracy_3_of_4‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.18595680594444275 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.2023809552192688 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.14057238399982452 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_6_of_7‚0 particle/accuracy_4_of_7‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.21495021879673004 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 jet/accuracy_1_of_7‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2635950446128845 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04454023391008377 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.20338983833789825 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 particle/accuracy_2_of_7‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1446090042591095 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 'particle/f_score‚0.3321704566478729 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.11555555462837219 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14057311415672302 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,jet/accuracy_3_of_6‚0.057692307978868484 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3031249940395355 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9885011911392212 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1459486037492752 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.20201146602630615 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0972222313284874 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.2395714521408081 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.1975308656692505 particle/accuracy_1_of_4‚1 particle/accuracy_1_of_5‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4092702567577362 -jet/accuracy_4_of_4‚0.0009920635493472219 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.375 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.2156156599521637 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4320167899131775 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13790971040725708 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.08315728604793549 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.24464288353919983 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3194444477558136 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 )jet/accuracy_1_of_1‚0.341918408870697 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.2081337422132492 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.011904762126505375 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.003500203834846616 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.006235827691853046 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_5‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.16741284728050232 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.21500132977962494 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.21001777052879333 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04399823769927025 particle/accuracy_1_of_1‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.3984259068965912 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13752378523349762 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.36006835103034973 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.2625174820423126 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1882360577583313 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.32499998807907104 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.309027761220932 *jet/accuracy_1_of_6‚0.8910256624221802 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08726852387189865 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.23472224175930023 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 _step‚197 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4962027668952942 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9925066232681274 6 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.111111119389534 ,jet/accuracy_3_of_4‚0.032680198550224304 *jet/accuracy_1_of_5‚0.8166539072990417 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.26524874567985535 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.09996459633111954 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.0833333358168602 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.25567764043807983 6 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.209196075797081 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 particle/accuracy_2_of_5‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 % _timestamp‚1.7884091902333477e+09 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16011983156204224 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.20025849342346191 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900468587875366 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3037280738353729 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.35879629850387573 particle/accuracy_5_of_6‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.10164283215999603 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.550000011920929 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.21169565618038177 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.39201605319976807 particle/accuracy_5_of_5‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3199999928474426 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 trainer/global_step‚2417 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.11108061671257019 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.26212167739868164 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.26870065927505493 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.40408530831336975 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9909008741378784 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.26655325293540955 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.1458333283662796 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.16831354796886444 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2265331894159317 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.15833333134651184 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 6 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.103448286652565 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_7_of_7‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.36934366822242737 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.24051563441753387 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17284747958183289 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.21662481129169464 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.007956549525260925 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.33763739466667175 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.006966649554669857 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3347596824169159 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.10112474858760834 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.022832242771983147 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.29629629850387573 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17480002343654633 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.44028419256210327 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.08160265535116196 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.26584821939468384 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16939088702201843 +jet/accuracy_2_of_2‚0.07921116799116135 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.1849086880683899 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.42752814292907715 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9892454147338867 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1596117913722992 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,jet/accuracy_3_of_5‚0.050358422100543976 particle/accuracy_3_of_3‚1 particle/accuracy_2_of_4‚1 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.1794973611831665 'particle/accuracy‚0.838500440120697 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03144654259085655 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.00382082792930305 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9975579977035522 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.19462814927101135 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.265625 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.20132488012313843 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0038746020291000605 +jet/accuracy_2_of_3‚0.18484266102313995 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.20372425019741058 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.2156156599521637 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2053571492433548 +particle/specificity‚0.9460254311561584 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9978796243667603 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.256729394197464 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.015796637162566185 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.2537330985069275 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.18518516421318054 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.011904762126505375 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6111111044883728 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.24795906245708466 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.29769957065582275 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,particle/sensitivity‚0.25744056701660156 particle/accuracy_3_of_6‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9962522387504578 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.40625 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9882514476776123 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.125 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.30000001192092896 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.1820702701807022 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.07280120253562927 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 7validation_average_jet_accuracy‚0.32777780294418335 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.2458639144897461 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.1928160935640335 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.00692239822819829 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.08484699577093124 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3778073489665985 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_4_of_5‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚56.99307222514535 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5151515007019043 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.16995464265346527 particle/accuracy_1_of_2‚1 jet/accuracy_5_of_6‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1702127605676651 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9298245310783386 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3856906592845917 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.5972180366516113 jet/accuracy_4_of_6‚0 particle/accuracy_6_of_6‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6425621509552002 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9841716885566711 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.18209876120090485 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_6‚0.33974361419677734 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1669503003358841 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.25057318806648254 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.23173238337039948 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3860960900783539 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_4_of_4‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18462078273296356 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9833787083625793 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9647988080978394 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.05367966368794441 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.1666666716337204 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9338061809539795 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9958333373069763 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.24425287544727325 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 epoch‚1 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 6 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.302325576543808 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.13963963091373444 particle/accuracy_2_of_6‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986336827278137 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.17573584616184235 ("Purity/3FRt4SRqqt0FBt/event_purity‚¥‰}Ìò0 particle/accuracy_5_of_7‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.1458333283662796 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9881753921508789 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.45575061440467834 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.007921649143099785 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.24074074625968933 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 _runtime‚1800.648542025 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.7043626308441162 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9747664332389832 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3670634627342224 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.2800000011920929 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9969736933708191 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9925458431243896 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7"Purity/*FRt0SRqqt*FBt/event_purity‚0.33919557929039 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.19342045485973358 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16287237405776978 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1858585923910141 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.15922456979751587 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3415229916572571 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.10299309343099594 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2531985342502594 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15628190338611603 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1826576590538025 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.11919737607240677 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_6‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.21052631735801697 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.1699240356683731 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 *jet/accuracy_1_of_4‚0.7813287377357483 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.2813165485858917 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.27005478739738464 jet/accuracy_7_of_7‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.11245917528867722 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.19364793598651886 particle/accuracy_1_of_3‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19681648910045624 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1376984566450119 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.26657429337501526 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365078806877136 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.006699909456074238 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.1701527237892151 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6341173648834229 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21666666865348816 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0Å‘‰ ‘·‚(:† ôêãÔøõž™ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.437' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8874 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ý4ࡸ‚(: ôêãÔ×ÜÖ›$ disk./.usageGB‚1057.273826599121 disk.sda3.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 463.9453125) proc.memory.percent‚17.67125129699707( disk./.usagePercent‚70.9545336148952 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1777.out‚0 disk.dm-1.out‚ 78.48046875 disk.rbd0.out‚0 disk.rbd2.out‚42.22216796875 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.982421875 disk.rbd2.in‚0$ memory_percent‚98.45867156982422% proc.memory.availableMB‚ 252.53125" proc.memory.rssMB‚ 2895.2578125 cpu‚25.4019185385934 network.recv‚703399 disk.sda.out‚ 78.48046875 disk.drbd1777.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 network.sent‚ 4.671571e+06 disk.sda.in‚0 disk.sda1.in‚0% disk.nvme0n1.out‚8033.95751953125 disk.rbd3.in‚0' disk.nvme0c0n1.out‚8033.95751953125 disk.sda3.out‚ 78.48046875 disk.dm-0.in‚ 463.9453125" disk.dm-0.out‚8033.95751953125 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 463.9453125{?žj2¹- _wandbruntime‚1825  _runtime‚1825P\ǺÁ & lr-AdamW/pg1‚9.783165970314063e-05 & lr-AdamW/pg2‚9.783165970314063e-05 trainer/global_step‚2449 % _timestamp‚1.7884092147088354e+09 _runtime‚1825.120965765 _step‚198¥ÉÁªÌ»Æ _runtime‚1825.120965765 _step‚198 & lr-AdamW/pg1‚9.783165970314063e-05 & lr-AdamW/pg2‚9.783165970314063e-05 trainer/global_step‚2449 % _timestamp‚1.7884092147088354e+09Æç¦5µ2¼- _wandbruntime‚1825  _runtime‚1825”ÞÆ'½ÀO XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚72656974363136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚73317961509888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3447382336665 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49571439 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚69199602216960 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49591476 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚72656787360768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚65826269449216 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚1825.121901759 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73317985449984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚69199602216960 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18900856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚502 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32373027 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3517502375424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876544 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚72656787360768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32373027 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚76835463885312 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚69199620219904 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3447217215265 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3457185143808 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3457354143232 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76835704701952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚69199620219904 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚69273469625121 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32373029 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3517719251968 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408I$¢HË'¾ÅO IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49591476 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚72656787360768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚65826269449216 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚1825.121901759 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73317985449984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚69199602216960 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18900856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚502 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32373027 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3517502375424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876544 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚72656787360768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32373027 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚76835463885312 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚69199620219904 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3447217215265 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3457185143808 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3457354143232 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76835704701952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚69199620219904 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚69273469625121 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32373029 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3517719251968 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚72656974363136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚73317961509888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3447382336665 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49571439 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚69199602216960Ç“Ѭÿ2¿- _wandbruntime‚1825  _runtime‚1825¡_[ÀËM R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚69300801972001 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81996977 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _step‚200 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32385688 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49611289 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238706176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚72685636879360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32385688 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚72685449863680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3458747294720 % _timestamp‚1.7884092153611777e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3458747294720 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚73346874458112 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3448771501417 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚484 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32954997 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 trainer/global_step‚2449 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3458578281984 VIDeviceStatsÚÒ¿vMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18908233 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32385686 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3448606379809 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49591226 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81976912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81996977 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3518930462720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚72685636879360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚69226871581696 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49591226 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81976912 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚69226889584640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18908231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚69226889584640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76866043627008 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚1825.774078735 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49611289 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14046766 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3519145228800 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32385686 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14047248 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚65852212631552 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚69226871581696 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32954997 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 trainer/global_step‚2449 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3458578281984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18908233 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32385686 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3448606379809 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49591226 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81976912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81996977 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3518930462720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚72685636879360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚69226871581696 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49591226 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81976912 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚69226889584640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18908231 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚69226889584640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76866043627008 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚1825.774078735 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49611289 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14046766 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3519145228800 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32385686 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14047248 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚65852212631552 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚69226871581696 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81996977 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _step‚200 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32385688 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49611289 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238706176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚72685636879360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32385688 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224Èà<=ü2Â- _wandbruntime‚1825  _runtime‚1825š¶ÈSÃÍ .loss/FRt1/detection_loss‚2.824262116406415 1loss/SRqqt4/detection_loss‚2.5005555550257363 .loss/FRt2/detection_loss‚2.476748188125326 .loss/FBt1/detection_loss‚5.519218146800995 -loss/FBt4/detection_loss‚4.05258604654899 1loss/FBt3/assignment_loss‚0.27143894691363757 /loss/FRt3/detection_loss‚2.4058374166488647 1loss/SRqqt1/detection_loss‚2.7553297515426363 0loss/SRqqt2/detection_loss‚1.796396170621333 _step‚201 0loss/FRt2/assignment_loss‚3.9212247843438006 /loss/FRt4/assignment_loss‚4.270102878411611 1loss/SRqqt3/assignment_loss‚1.916943615979769 1loss/SRqqt3/detection_loss‚2.0909617358649317 % _timestamp‚1.7884092153631463e+09 _runtime‚1825.775334782 &loss/total_loss‚58.685242045996354 0loss/FRt3/assignment_loss‚2.7371994504114476 1loss/SRqqt2/assignment_loss‚1.526343228538399 /loss/FRt4/detection_loss‚2.2182470526960163 .loss/FBt3/detection_loss‚1.025842910406219 trainer/global_step‚2449 /loss/FRt1/assignment_loss‚4.493764719447574 1loss/SRqqt1/assignment_loss‚2.346275211444923 1loss/SRqqt4/assignment_loss‚1.681632092429532 epoch‚2 0loss/FBt1/assignment_loss‚0.4260881293924355 -loss/FBt2/detection_loss‚4.65912418067525 0loss/FBt2/assignment_loss‚0.4309855067403987 1loss/FBt4/assignment_loss‚0.338134210580112564C»MXÄÒ 0loss/FRt2/assignment_loss‚3.9212247843438006 /loss/FRt4/assignment_loss‚4.270102878411611 1loss/SRqqt3/assignment_loss‚1.916943615979769 1loss/SRqqt3/detection_loss‚2.0909617358649317 % _timestamp‚1.7884092153631463e+09 _runtime‚1825.775334782 &loss/total_loss‚58.685242045996354 0loss/FRt3/assignment_loss‚2.7371994504114476 1loss/SRqqt2/assignment_loss‚1.526343228538399 /loss/FRt4/detection_loss‚2.2182470526960163 .loss/FBt3/detection_loss‚1.025842910406219 trainer/global_step‚2449 /loss/FRt1/assignment_loss‚4.493764719447574 1loss/SRqqt1/assignment_loss‚2.346275211444923 1loss/SRqqt4/assignment_loss‚1.681632092429532 epoch‚2 0loss/FBt1/assignment_loss‚0.4260881293924355 -loss/FBt2/detection_loss‚4.65912418067525 0loss/FBt2/assignment_loss‚0.4309855067403987 1loss/FBt4/assignment_loss‚0.33813421058011256 .loss/FRt1/detection_loss‚2.824262116406415 1loss/SRqqt4/detection_loss‚2.5005555550257363 .loss/FRt2/detection_loss‚2.476748188125326 .loss/FBt1/detection_loss‚5.519218146800995 -loss/FBt4/detection_loss‚4.05258604654899 1loss/FBt3/assignment_loss‚0.27143894691363757 /loss/FRt3/detection_loss‚2.4058374166488647 1loss/SRqqt1/detection_loss‚2.7553297515426363 0loss/SRqqt2/detection_loss‚1.796396170621333 _step‚201Éü-`hÅ‚(:… ƒëãÔ‹¯º™ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.353' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0706 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a€}½Æ‚(:’ ƒëãÔÆÿóœ disk.sda3.out‚ 79.1171875 disk.drbd1777.out‚0 disk.rbd0.in‚0 disk.rbd2.out‚43.79638671875 disk.sda2.in‚0" proc.memory.rssMB‚ 2895.2578125) proc.memory.percent‚17.67125129699707 disk.rbd1.in‚0 disk.rbd1.out‚ 0.982421875 disk.rbd2.in‚0( proc.memory.availableMB‚ 252.74609375$ disk./.usageGB‚1057.273349761963 disk.sda1.in‚0 disk.rbd3.in‚0& disk.nvme0c0n1.out‚8141.7939453125 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda.in‚0 network.sent‚ 4.795001e+06 network.recv‚709454 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 79.1171875$ memory_percent‚98.45736026763916 disk.drbd1777.in‚0 disk.dm-0.in‚ 466.15234375 cpu‚25.39903727075504 proc.cpu.threads‚17$ disk.nvme0n1.out‚8141.7939453125 disk.rbd0.out‚0 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 466.15234375 disk.sda.out‚ 79.1171875! disk.dm-0.out‚8141.7939453125) disk./.usagePercent‚70.95450161395341 disk.nvme0n1.in‚ 466.15234375øQ–œÇ‚(:‘ ’ëãÔ󱊛 gpu.0.gpu‚20.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.389' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.677799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„¯b~È‚(:” ’ëãÔ‘¥”œ disk.nvme0n1.in‚ 466.70703125 disk.dm-0.in‚ 466.70703125 disk.rbd2.out‚43.81201171875) disk./.usagePercent‚70.95448215738081 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd2.in‚0 cpu‚25.396258004348457 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 466.70703125 disk.sda2.in‚0 disk.drbd1777.in‚0! disk.dm-0.out‚8257.4189453125 disk.rbd0.in‚0( proc.memory.availableMB‚ 252.98046875" proc.memory.rssMB‚ 2895.2578125 disk.sda.out‚ 79.4609375 disk.sda1.in‚0$ memory_percent‚98.45592975616455 network.sent‚ 4.903663e+06& disk.nvme0c0n1.out‚8257.4189453125 disk.rbd1.out‚ 0.982421875% disk./.usageGB‚1057.2730598449707 disk.rbd3.out‚0 disk.sda3.out‚ 79.4609375 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd1.in‚0) proc.memory.percent‚17.67125129699707 network.recv‚716601 disk.sda2.out‚0$ disk.nvme0n1.out‚8257.4189453125 disk.dm-1.out‚ 79.4609375 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1777.out‚0ltدœÉ‚(:‘ ¡ëãÔìƒðš gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.066' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.813200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª£‚Ê‚(:‹ ¡ëãÔ «’œ$ disk.nvme0n1.out‚8356.6005859375 network.recv‚720707 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.sda3.in‚0 cpu‚25.39363500366857 network.sent‚ 4.909724e+06) disk./.usagePercent‚70.95448215738081 disk.sda.in‚0! disk.dm-0.out‚8356.6005859375 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 466.70703125 disk.dm-0.in‚ 466.70703125 disk.rbd0.in‚0 disk.sda.out‚ 80.10546875 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sda3.out‚ 80.10546875! proc.memory.availableMB‚252.5 disk.rbd1.out‚ 0.982421875 disk.nvme0n1.in‚ 466.70703125 disk.dm-1.out‚ 80.10546875# memory_percent‚98.4588623046875) proc.memory.percent‚17.67125129699707& disk.nvme0c0n1.out‚8356.6005859375% disk./.usageGB‚1057.2730598449707 disk.rbd2.out‚ 43.96484375 disk.rbd3.in‚0" proc.memory.rssMB‚ 2895.2578125 proc.cpu.threads‚17›÷ÿ2Ë- _wandbruntime‚1860  _runtime‚1860–R£·ÅÌ¿ _runtime‚1860.490501698 _step‚202 % lr-AdamW/pg1‚9.75711877699702e-05 % lr-AdamW/pg2‚9.75711877699702e-05 trainer/global_step‚2499 % _timestamp‚1.7884092500783005e+09Þ¿ÊÍÄ % lr-AdamW/pg1‚9.75711877699702e-05 % lr-AdamW/pg2‚9.75711877699702e-05 trainer/global_step‚2499 % _timestamp‚1.7884092500783005e+09 _runtime‚1860.490501698 _step‚202Ê^\ 2Î- _wandbruntime‚1860  _runtime‚18604õ¿Z$ÏÁO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚490 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 % _timestamp‚1.7884092500791094e+09 _step‚203 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚70640100560329 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50564141 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74089926920192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14313745 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3526856466432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234525184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67123411526656 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50584178 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33005977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚70563070453760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83570118 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚70640282721089 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33005979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19269706 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210585088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33582959 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33005979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3516689033673 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74089926920192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14313745 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3526856466432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234525184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67123411526656 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50584178 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33005977 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚70563070453760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83570118 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚70640282721089 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33005979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19269706 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210585088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33582959 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33005979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3516689033673 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83602564 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50603991 _step‚204 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚74118589423104 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚70590357821440 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 _runtime‚1861.163166761 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33596074 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14318991 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚78382660325888 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚67149371748352 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3528418617344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3528249604608 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74792521869312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚70590357821440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83622629 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33018636 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚78382430008320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33595576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19277083 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83602564 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3590114516480 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚67149354708992 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3518243319825 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 trainer/global_step‚2499 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3528249604608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50583928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206377472 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3528418617344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33018638 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70667615068177 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚496 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50583928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33595576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19277083 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83602564 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3590114516480 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚67149354708992 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3518243319825 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 trainer/global_step‚2499 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3528249604608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50583928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206377472 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3528418617344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33018638 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70667615068177 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚496 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50583928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83602564 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50603991 _step‚204 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚74118589423104 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚70590357821440 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 _runtime‚1861.163166761 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33596074 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14318991 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚78382660325888 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚67149371748352 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3528418617344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3528249604608 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74792521869312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚70590357821440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83622629 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33018636 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚78382430008320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3659447299072 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3586160852081 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85195770 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚75523253481984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34223148 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33638929 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51556843 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚75523066479616 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33638927 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51556843 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233728 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚76209280272384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3596696788480 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚75523066479616 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚79868931865088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51576880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚68420587682816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚71926556693504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3596527789056 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51576880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14584067 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3586325973481 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85215809 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚71926538690560 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34222621 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76209256332288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚72006913656297 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚1895.982345335 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚72006731495537 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚75523253481984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3596696788480 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19638554 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14584592 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204293632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7884092855698597e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0jÉ^Ì'àÆO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19638554 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14584592 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204293632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7884092855698597e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3659447299072 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3586160852081 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85195770 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚75523253481984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34223148 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33638929 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51556843 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚75523066479616 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33638927 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51556843 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233728 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚76209280272384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3596696788480 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚75523066479616 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚79868931865088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51576880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚68420587682816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚71926556693504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3596527789056 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51576880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14584067 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3586325973481 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85215809 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚71926538690560 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34222621 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76209256332288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚72006913656297 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚1895.982345335 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚72006731495537 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚75523253481984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3596696788480 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965Ï»×2á- _wandbruntime‚1896  _runtime‚1896è[Zz_âËM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _runtime‚1896.649362001 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚507 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚75551728982528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3598089939968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19645933 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34235341 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51576630 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚505 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33651586 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85248281 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _step‚208 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3597920927232 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 % _timestamp‚1.7884092862364037e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚68446513825792 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3597920927232 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85248281 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3598089939968 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34235848 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206377472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚71953826058240 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79899266316800 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51596693 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚75551915998208 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚71953826058240 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚76238193220608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34235341 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51576630 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚505 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33651586 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85248281 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _step‚208 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3597920927232 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 % _timestamp‚1.7884092862364037e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚68446513825792 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3597920927232 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85248281 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3598089939968 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34235848 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206377472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚71953826058240 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79899266316800 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51596693 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0Ð"¹2ä- _wandbruntime‚1896  _runtime‚1896ðA®ÔVåÐ 0loss/FRt4/assignment_loss‚3.7648687627580433 2loss/SRqqt3/assignment_loss‚2.1828417372508127 /loss/SRqqt1/detection_loss‚2.66813562810421 /loss/FBt2/detection_loss‚2.2756714522838593 /loss/FBt3/detection_loss‚1.3450107297950376 %loss/total_loss‚55.46079650268749 2loss/SRqqt2/assignment_loss‚2.1149058143826243 /loss/FBt1/detection_loss‚3.7235847523337915 % _timestamp‚1.7884092862382119e+09 /loss/FRt1/assignment_loss‚6.154006949177495 2loss/SRqqt4/assignment_loss‚1.6962696216229736 0loss/FBt4/assignment_loss‚0.4517374099438409 1loss/SRqqt2/detection_loss‚1.3952886365512585 _step‚209 0loss/FRt3/assignment_loss‚2.8442552945552726 0loss/FBt2/assignment_loss‚0.5727358981966972 /loss/FRt2/detection_loss‚2.1332077168185135 -loss/FRt4/detection_loss‚2.32165458533499 trainer/global_step‚2549 /loss/FRt1/detection_loss‚3.2393374111917286 0loss/SRqqt3/detection_loss‚1.879732836709648 0loss/SRqqt4/detection_loss‚1.716016435931469 /loss/FBt4/detection_loss‚3.8325140327215195 epoch‚2 _runtime‚1896.650306352 1loss/SRqqt1/assignment_loss‚2.733218272527059 0loss/FBt1/assignment_loss‚0.4161194614398004 0loss/FBt3/assignment_loss‚0.3339368286006738 .loss/FRt3/detection_loss‚2.392837244730729 0loss/FRt2/assignment_loss‚3.2729089897254418Sâ°û[æÕ .loss/FRt3/detection_loss‚2.392837244730729 0loss/FRt2/assignment_loss‚3.2729089897254418 0loss/FRt4/assignment_loss‚3.7648687627580433 2loss/SRqqt3/assignment_loss‚2.1828417372508127 /loss/SRqqt1/detection_loss‚2.66813562810421 /loss/FBt2/detection_loss‚2.2756714522838593 /loss/FBt3/detection_loss‚1.3450107297950376 %loss/total_loss‚55.46079650268749 2loss/SRqqt2/assignment_loss‚2.1149058143826243 /loss/FBt1/detection_loss‚3.7235847523337915 % _timestamp‚1.7884092862382119e+09 /loss/FRt1/assignment_loss‚6.154006949177495 2loss/SRqqt4/assignment_loss‚1.6962696216229736 0loss/FBt4/assignment_loss‚0.4517374099438409 1loss/SRqqt2/detection_loss‚1.3952886365512585 _step‚209 0loss/FRt3/assignment_loss‚2.8442552945552726 0loss/FBt2/assignment_loss‚0.5727358981966972 /loss/FRt2/detection_loss‚2.1332077168185135 -loss/FRt4/detection_loss‚2.32165458533499 trainer/global_step‚2549 /loss/FRt1/detection_loss‚3.2393374111917286 0loss/SRqqt3/detection_loss‚1.879732836709648 0loss/SRqqt4/detection_loss‚1.716016435931469 /loss/FBt4/detection_loss‚3.8325140327215195 epoch‚2 _runtime‚1896.650306352 1loss/SRqqt1/assignment_loss‚2.733218272527059 0loss/FBt1/assignment_loss‚0.4161194614398004 0loss/FBt3/assignment_loss‚0.3339368286006738ÑR'5…‘ç‚(:† ÎëãÔ‘”ƒš gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.857' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5714 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó5Nn"è‚(:— ÎëãÔ‹çÕ› disk.sda3.out‚ 81.59765625 disk.sda.in‚0 disk.sda.out‚ 81.59765625$ memory_percent‚98.45778942108154% disk./.usageGB‚1057.2730598449707 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 470.55078125 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 470.55078125% disk.nvme0n1.out‚8556.05908203125" disk.dm-0.out‚8556.05908203125 disk.dm-1.out‚ 81.59765625( proc.memory.availableMB‚ 252.67578125 cpu‚25.385634420364838 network.recv‚734909 disk.drbd1777.in‚0 disk.dm-1.in‚0) proc.memory.percent‚17.67125129699707 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.130859375 disk.rbd2.out‚ 45.56640625 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 470.55078125 network.sent‚ 5.028515e+06 disk.rbd2.in‚0' disk.nvme0c0n1.out‚8556.05908203125 disk.sda1.in‚0" proc.memory.rssMB‚ 2895.2578125 proc.cpu.threads‚17) disk./.usagePercent‚70.95448215738081 disk.sda2.in‚0 disk.sda3.in‚0Ø#«Ï‘é‚(:† ÝëãÔÙ‚²™ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.027' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07ïaê‚(:ˆ ÝëãÔÊÏøœ disk.dm-1.out‚82.15625 disk.rbd1.out‚ 1.1630859375 disk.rbd2.out‚ 45.62890625 disk.nvme0c0n1.in‚ 518.140625 network.recv‚741508 disk.rbd0.out‚0" proc.memory.rssMB‚ 2895.2578125) disk./.usagePercent‚70.95448215738081% disk.nvme0n1.out‚8655.12158203125" disk.dm-0.out‚8655.12158203125 disk.nvme0n1.in‚ 518.140625 disk.dm-0.in‚ 518.140625 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda.out‚82.15625 disk.sda.in‚0 network.sent‚ 5.137207e+06 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.sda3.out‚82.15625 disk.sda1.in‚0# memory_percent‚98.4595537185669 cpu‚25.383010964904816% disk./.usageGB‚1057.2730598449707 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0' disk.nvme0c0n1.out‚8655.12158203125 disk.sda3.in‚0( proc.memory.availableMB‚ 252.38671875 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda1.out‚0) proc.memory.percent‚17.67125129699707>S'á2ë- _wandbruntime‚1931  _runtime‚1931ž í Åì¿ trainer/global_step‚2599 % _timestamp‚1.7884093209950182e+09 _runtime‚1931.407216065 _step‚210 % lr-AdamW/pg1‚9.70208824048177e-05 % lr-AdamW/pg2‚9.70208824048177e-05,r"ÊíÄ trainer/global_step‚2599 % _timestamp‚1.7884093209950182e+09 _runtime‚1931.407216065 _step‚210 % lr-AdamW/pg1‚9.70208824048177e-05 % lr-AdamW/pg2‚9.70208824048177e-05Ңʌ>2î- _wandbruntime‚1931  _runtime‚1931¨(Ç'ïÁO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚69717746799616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3655797791889 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7884093209957895e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52549545 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76956206039040 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86841461 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14855965 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86821422 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3730556723200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3655632670489 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3666368111104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚73373362430745 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3666368111104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20007404 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127j•ìÿ[ðÆO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76956393041408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14855451 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3730348235264 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2599 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3655797791889 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7884093209957895e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52549545 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76956206039040 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86841461 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14855965 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86821422 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3730556723200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3655632670489 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3666368111104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚73373362430745 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3666368111104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20007404 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚69717746799616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 BYþÑ&òËM _runtime‚1932.068554794 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚73317294295040 trainer/global_step‚2599 % _timestamp‚1.7884093216558967e+09 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚73400694777625 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76985055557632 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52589395 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52589395 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚73317276292096 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86873933 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚73317294295040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚512 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3731974185472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14861351 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3667761262592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚510 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3667592249856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20014783 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76984868541952 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚76985055557632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34284538 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52569332 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414720 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚69743689981952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚77683816691712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3731765710848 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52569332 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚73317276292096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14860841 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34875622 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3667592249856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚81415814817280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34284536 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86873933 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20014781 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3657186956641 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚69743672942592 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0hž¹²Ö&óÐM R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚73400694777625 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76985055557632 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52589395 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52589395 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚73317276292096 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86873933 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚73317294295040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚512 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3731974185472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14861351 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3667761262592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚510 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3667592249856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20014783 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76984868541952 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚76985055557632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34284538 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52569332 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414720 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚69743689981952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚77683816691712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3731765710848 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52569332 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚73317276292096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14860841 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34875622 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3667592249856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚81415814817280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34284536 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86873933 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20014781 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3657186956641 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚69743672942592 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚1932.068554794 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚73317294295040 trainer/global_step‚2599 % _timestamp‚1.7884093216558967e+09ÔÖ?µô2ô- _wandbruntime‚1932  _runtime‚1932EøÝUõÏ _runtime‚ 1932.06997508 1loss/SRqqt3/assignment_loss‚2.002838031972869 2loss/SRqqt4/assignment_loss‚1.7129047389252712 0loss/FBt4/assignment_loss‚0.5381104064435931 .loss/FBt2/detection_loss‚2.416779577732086 _step‚213 trainer/global_step‚2599 $ _timestamp‚1.788409321657917e+09 /loss/FRt1/detection_loss‚3.1036814046757564 .loss/FRt4/detection_loss‚3.107540353681102 1loss/SRqqt2/detection_loss‚1.5252149347941597 0loss/SRqqt3/detection_loss‚1.572324640543784 %loss/total_loss‚56.20381350375307 /loss/FRt2/assignment_loss‚3.864157971213846 /loss/FRt4/assignment_loss‚3.521560575022842 1loss/FBt1/assignment_loss‚0.36752893227081324 0loss/FBt2/assignment_loss‚0.5397596564143896 0loss/FBt3/assignment_loss‚0.2554850939064579 /loss/FBt3/detection_loss‚0.9917136556218462 1loss/SRqqt1/detection_loss‚2.9298217617548428 .loss/FBt1/detection_loss‚4.537072828837803 1loss/SRqqt1/assignment_loss‚2.391842782497406 /loss/FRt3/detection_loss‚2.0053861335567804 /loss/FBt4/detection_loss‚4.4710867371824055 2loss/SRqqt2/assignment_loss‚1.9870771052581924 /loss/FRt2/detection_loss‚2.4087329372471453 1loss/SRqqt4/detection_loss‚1.7173117601265342 0loss/FRt3/assignment_loss‚2.7239745622095852 /loss/FRt1/assignment_loss‚5.511906921863556 epoch‚2°Y^ZöÔ /loss/FRt4/assignment_loss‚3.521560575022842 1loss/FBt1/assignment_loss‚0.36752893227081324 0loss/FBt2/assignment_loss‚0.5397596564143896 0loss/FBt3/assignment_loss‚0.2554850939064579 /loss/FBt3/detection_loss‚0.9917136556218462 1loss/SRqqt1/detection_loss‚2.9298217617548428 .loss/FBt1/detection_loss‚4.537072828837803 1loss/SRqqt1/assignment_loss‚2.391842782497406 /loss/FRt3/detection_loss‚2.0053861335567804 /loss/FBt4/detection_loss‚4.4710867371824055 2loss/SRqqt2/assignment_loss‚1.9870771052581924 /loss/FRt2/detection_loss‚2.4087329372471453 1loss/SRqqt4/detection_loss‚1.7173117601265342 0loss/FRt3/assignment_loss‚2.7239745622095852 /loss/FRt1/assignment_loss‚5.511906921863556 epoch‚2 _runtime‚ 1932.06997508 1loss/SRqqt3/assignment_loss‚2.002838031972869 2loss/SRqqt4/assignment_loss‚1.7129047389252712 0loss/FBt4/assignment_loss‚0.5381104064435931 .loss/FBt2/detection_loss‚2.416779577732086 _step‚213 trainer/global_step‚2599 $ _timestamp‚1.788409321657917e+09 /loss/FRt1/detection_loss‚3.1036814046757564 .loss/FRt4/detection_loss‚3.107540353681102 1loss/SRqqt2/detection_loss‚1.5252149347941597 0loss/SRqqt3/detection_loss‚1.572324640543784 %loss/total_loss‚56.20381350375307 /loss/FRt2/assignment_loss‚3.864157971213846Õ–¬Ý÷‚(:… ìëãÔà¤Í™ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.495' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.299 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Uèe ø‚(:ˆ ìëãԔϜ disk.drbd1777.in‚0 disk.nvme0n1.in‚ 661.28125 disk.dm-0.in‚ 661.28125 network.sent‚ 5.142519e+06% disk./.usageGB‚1057.2744483947754 disk.sda1.out‚0 disk.rbd1.out‚ 1.1630859375) proc.memory.percent‚17.67125129699707 network.recv‚745112 disk.nvme0c0n1.in‚ 661.28125 disk.sda.out‚ 82.98046875 disk.drbd1777.out‚0 disk.sda3.out‚ 82.98046875 disk.rbd3.out‚0 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.dm-0.out‚8656.466796875$ memory_percent‚98.45938682556152) disk./.usagePercent‚70.95457534412328% disk.nvme0c0n1.out‚8656.466796875 disk.sda3.in‚0 disk.dm-1.out‚ 82.98046875 disk.rbd0.in‚0' proc.memory.availableMB‚ 252.4140625" proc.memory.rssMB‚ 2895.2578125 cpu‚25.380598074529978 disk.rbd2.out‚ 45.64453125 disk.sda2.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0# disk.nvme0n1.out‚8656.466796875 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda.in‚0œlAµ›ù‚(: ûëãÔ–Á¡™ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚89.096' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.819200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ™{Òú‚(:Ž ûëãÔÀÕ‹œ disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 83.703125" proc.memory.rssMB‚ 2895.2578125 cpu‚25.37797425968466 disk.sda.out‚ 83.703125 network.recv‚751280 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.sda.in‚0) proc.memory.percent‚17.67125129699707 network.sent‚ 5.25062e+06" disk.dm-0.out‚8755.62451171875 disk.rbd0.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1057.2749404907227 disk.dm-1.out‚ 83.703125 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.22607421875 disk.dm-0.in‚ 664.3828125( disk./.usagePercent‚70.9546083690952 disk.rbd0.in‚0 disk.rbd2.out‚ 45.7431640625! disk.nvme0c0n1.in‚ 664.3828125$ memory_percent‚98.45824241638184' proc.memory.availableMB‚ 252.6015625 disk.rbd3.out‚0' disk.nvme0c0n1.out‚8755.62451171875 disk.sda2.out‚0 disk.nvme0n1.in‚ 664.3828125% disk.nvme0n1.out‚8755.62451171875 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd1777.in‚0›çL¿‘û‚(:† ŠìãÔ‹×™™ gpu.0.gpu‚25.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.872' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b4 ’ü‚(:ö ŠìãÔÔïÏœ disk.rbd1.out‚ 1.22607421875 proc.cpu.threads‚17 network.sent‚ 5.25677e+06 network.recv‚756265 disk.drbd1777.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2895.2578125 cpu‚25.375613310980288 disk.rbd2.in‚0 disk.rbd2.out‚ 45.7587890625 disk.sda2.in‚0 disk.sda3.in‚0' disk./.usagePercent‚70.954610673163 disk.drbd1777.out‚0 disk.dm-0.out‚8856.5 disk.dm-1.in‚0 disk.dm-0.in‚ 664.3828125 disk.sda.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0$ memory_percent‚98.45831394195557 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 664.3828125 disk.sda1.in‚0 ŒKØodisk.nvme0n1.in‚ 664.3828125 disk.rbd0.out‚0 disk.sda3.out‚ 84.32421875 disk.sda.out‚ 84.32421875 disk.rbd3.out‚0$ disk./.usageGB‚1057.274974822998 disk.rbd1.in‚0 disk.nvme0n1.out‚8856.5( proc.memory.availableMB‚ 252.58984375) proc.memory.percent‚17.67125129699707 disk.dm-1.out‚ 84.32421875 disk.nvme0c0n1.out‚8856.5û!Þc2ý- _wandbruntime‚1966  _runtime‚1966'1+ŠÅþ¿ trainer/global_step‚2649 % _timestamp‚1.7884093565269043e+09 _runtime‚1966.939089351 _step‚214 % lr-AdamW/pg1‚9.67311636556489e-05 % lr-AdamW/pg2‚9.67311636556489e-05rÔ_ÊÿÄ % lr-AdamW/pg1‚9.67311636556489e-05 % lr-AdamW/pg2‚9.67311636556489e-05 trainer/global_step‚2649 % _timestamp‚1.7884093565269043e+09 _runtime‚1966.939089351 _step‚214ÖT þ]2€- _wandbruntime‚1966  _runtime‚1966%î…Å'¿O _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚74653493167104 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53542247 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79100575094784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚74653493167104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3801507077632 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 # _timestamp‚1.78840935652761e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚78389532600832 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3735870434304 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚74653475164160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚71014905916416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚79100551154688 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210585088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34904827 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3736039433728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15126202 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34904827 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3735870434304 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78389345598464 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚544 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53562284 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20376254 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3725269610297 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3725104488897 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15126746 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚74740175526713 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34904829 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚1966.939942428 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚78389532600832 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚546 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234525184 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3801296492544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88447074 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20376256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88467113 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53542247 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚82902082172416 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82901847647232 trainer/global_step‚2649 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚215 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34904829 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚74653475164160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944ÕPsÊ'‚ÄO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88467113 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53542247 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚82902082172416 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82901847647232 trainer/global_step‚2649 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚215 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34904829 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚74653475164160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚74653493167104 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53542247 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79100575094784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚74653493167104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3801507077632 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 # _timestamp‚1.78840935652761e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚78389532600832 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3735870434304 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚74653475164160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚71014905916416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚79100551154688 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210585088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34904827 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3736039433728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15126202 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34904827 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3735870434304 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78389345598464 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚544 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53562284 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20376254 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3725269610297 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3725104488897 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15126746 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚74740175526713 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34904829 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚1966.939942428 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚78389532600832 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚546 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234525184 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3801296492544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88447074 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20376256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408×ð\2ƒ- _wandbruntime‚1967  _runtime‚1967/qÑ&„ËM E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3737263572480 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35515188 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88499585 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34917488 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚78418195117056 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15132068 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88499585 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53582097 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15131557 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35515701 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚82932185185792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204280320 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34917486 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚78418195117056 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79129464102912 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚71040849098752 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 % _timestamp‚1.7884093571899538e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚74680744528896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚74767325712833 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3737432585216 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34917488 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚511 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚74680744528896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚74767507873801 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3726658775049 _step‚216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53562034 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53562034 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚71040832059392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82932413406208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚2649 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚78418008101376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88479520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20383631 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚79129488043008 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3802721082880 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚74680762531840 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34917486 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20383633 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 % _timestamp‚1.7884093571899538e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚74680744528896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚74767325712833 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3737432585216 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34917488 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚511 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚74680744528896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚74767507873801 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3726658775049 _step‚216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53562034 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_tra°t«‚!in_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53562034 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚71040832059392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82932413406208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚2649 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚78418008101376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88479520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20383631 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚79129488043008 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3802721082880 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚74680762531840 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34917486 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20383633 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35515188 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88499585 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34917488 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚78418195117056 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15132068 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88499585 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53582097 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15131557 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35515701 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚82932185185792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204280320 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34917486 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚78418195117056 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79129464102912 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚71040849098752 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199ØlŬÚ2†- _wandbruntime‚1967  _runtime‚1967¯Å÷ñW‡Ñ 0loss/SRqqt4/detection_loss‚2.060371670499444 trainer/global_step‚2649 $ _timestamp‚1.788409357192029e+09 2loss/SRqqt2/assignment_loss‚1.7819342671985356 2loss/SRqqt3/assignment_loss‚1.9272429990865614 /loss/FBt1/detection_loss‚3.8911320461946377 /loss/FBt3/detection_loss‚0.9788051193064947 .loss/FBt4/detection_loss‚3.296065640449524 /loss/FRt1/assignment_loss‚4.951056307759778 0loss/FBt4/assignment_loss‚0.4429227506960766 epoch‚2 _runtime‚ 1967.60407807 /loss/FRt2/assignment_loss‚3.291758636633555 2loss/SRqqt4/assignment_loss‚1.7669811263680457 1loss/SRqqt1/detection_loss‚2.7215302098881113 %loss/total_loss‚55.87327238598356 1loss/FBt1/assignment_loss‚0.37447215969950776 0loss/FBt2/assignment_loss‚0.5761389922869663 /loss/FRt4/detection_loss‚2.1203725528209767 0loss/FRt3/assignment_loss‚2.7257079395380885 /loss/FRt4/assignment_loss‚4.633632401202587 2loss/SRqqt1/assignment_loss‚2.3939883781201914 1loss/SRqqt2/detection_loss‚1.6850647057445955 1loss/SRqqt3/detection_loss‚1.9156314590540917 .loss/FBt2/detection_loss‚4.849622347702582 _step‚217 1loss/FBt3/assignment_loss‚0.20539334540571622 /loss/FRt1/detection_loss‚2.9763591762246757 .loss/FRt2/detection_loss‚2.330766805013021 .loss/FRt3/detection_loss‚1.976321349089796üüZú\ˆÖ /loss/FRt4/detection_loss‚2.1203725528209767 0loss/FRt3/assignment_loss‚2.7257079395380885 /loss/FRt4/assignment_loss‚4.633632401202587 2loss/SRqqt1/assignment_loss‚2.3939883781201914 1loss/SRqqt2/detection_loss‚1.6850647057445955 1loss/SRqqt3/detection_loss‚1.9156314590540917 .loss/FBt2/detection_loss‚4.849622347702582 _step‚217 1loss/FBt3/assignment_loss‚0.20539334540571622 /loss/FRt1/detection_loss‚2.9763591762246757 .loss/FRt2/detection_loss‚2.330766805013021 .loss/FRt3/detection_loss‚1.976321349089796 0loss/SRqqt4/detection_loss‚2.060371670499444 trainer/global_step‚2649 $ _timestamp‚1.788409357192029e+09 2loss/SRqqt2/assignment_loss‚1.7819342671985356 2loss/SRqqt3/assignment_loss‚1.9272429990865614 /loss/FBt1/detection_loss‚3.8911320461946377 /loss/FBt3/detection_loss‚0.9788051193064947 .loss/FBt4/detection_loss‚3.296065640449524 /loss/FRt1/assignment_loss‚4.951056307759778 0loss/FBt4/assignment_loss‚0.4429227506960766 epoch‚2 _runtime‚ 1967.60407807 /loss/FRt2/assignment_loss‚3.291758636633555 2loss/SRqqt4/assignment_loss‚1.7669811263680457 1loss/SRqqt1/detection_loss‚2.7215302098881113 %loss/total_loss‚55.87327238598356 1loss/FBt1/assignment_loss‚0.37447215969950776 0loss/FBt2/assignment_loss‚0.5761389922869663ÙÛg*€‘‰‚(:† ™ìãÔª‘ë™ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.294' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4588 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à Ž Š‚(:‚ ™ìãÔÖÒÒœ disk.rbd2.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 85.0390625 disk.drbd1777.in‚0 disk.rbd2.out‚ 47.3955078125 disk.rbd3.out‚0 cpu‚25.373175808596375 network.sent‚ 5.262843e+06 network.recv‚760510 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 708.1796875 disk.dm-1.in‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 128.578125 disk.sda3.in‚0 disk.rbd1.out‚ 1.2890625! disk.nvme0n1.out‚ 9023.0078125 disk.dm-1.out‚ 85.0390625 disk.sda2.in‚0 disk.rbd1.in‚0% disk./.usageGB‚1056.2111282348633 disk.sda3.out‚ 85.0390625! disk.nvme0c0n1.in‚ 708.1796875# disk.nvme0c0n1.out‚ 9023.0078125# memory_percent‚99.2152214050293" proc.memory.rssMB‚ 2895.2578125) disk./.usagePercent‚70.88321503600655 disk.rbd3.in‚0) proc.memory.percent‚17.67125129699707 proc.cpu.threads‚17 disk.nvme0n1.in‚ 708.1796875 disk.dm-0.out‚ 9023.0078125ŸØ–‰‘‹‚(:† ¨ìãÔ¾ú˜™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.419' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0838 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nP ÂŒ‚(: ¨ìãÔŽÇ©›) disk./.usagePercent‚70.88321657205174 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.rbd1.out‚ 1.2890625 disk.rbd2.out‚ 47.4111328125$ memory_percent‚99.21374320983887% disk./.usageGB‚1056.2111511230469& disk.nvme0c0n1.out‚9023.4404296875 disk.sda3.out‚ 85.4765625 disk.rbd0.in‚0 cpu‚25.370810311946226 network.recv‚767027 disk.sda.out‚ 85.4765625 disk.rbd1.in‚0! disk.dm-0.out‚9023.4404296875 disk.rbd2.in‚0) proc.memory.percent‚17.67125129699707 proc.cpu.threads‚17 disk.drbd1777.out‚0$ disk.nvme0n1.out‚9023.4404296875 disk.dm-1.out‚ 85.4765625' proc.memory.availableMB‚ 128.8203125" proc.memory.rssMB‚ 2895.2578125! disk.nvme0c0n1.in‚ 708.1796875 disk.sda.in‚0 disk.dm-0.in‚ 708.1796875 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 708.1796875 network.sent‚ 5.370847e+06 disk.sda3.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0J¶¿2- _wandbruntime‚2002  _runtime‚2002Ág,‡ÅŽ¿ % lr-AdamW/pg1‚9.64318140248079e-05 % lr-AdamW/pg2‚9.64318140248079e-05 trainer/global_step‚2699 % _timestamp‚1.7884093918143754e+09 _runtime‚2002.226511388 _step‚218@Š‰_ÊÄ _runtime‚2002.226511388 _step‚218 % lr-AdamW/pg1‚9.64318140248079e-05 % lr-AdamW/pg2‚9.64318140248079e-05 trainer/global_step‚2699 % _timestamp‚1.7884093918143754e+09ÚµžLé2- _wandbruntime‚2002  _runtime‚2002é„ŽøÆ'‘ÀO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54534949 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚80546222505984 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚79822672160256 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3872184661504 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚72312047993856 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3805710756352 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚79822485157888 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚79822672160256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚2002.227444556 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54554986 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3805541756928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35537779 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3872397343744 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚76016943400960 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3805710756352 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79822485157888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15397001 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚80546198565888 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2699 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54534949 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36142615 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3805541756928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682240 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚76106624301161 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚72312065033216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3794576307305 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚510 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35537779 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35537777 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚76016943400960 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚76016961403904 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0p÷g©’ÅO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3794576307305 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚510 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35537779 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35537777 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚76016943400960 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚76016961403904 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54534949 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚80546222505984 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚79822672160256 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3872184661504 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚72312047993856 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3805710756352 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚79822485157888 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚79822672160256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚2002.227444556 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54554986 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3805541756928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35537779 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3872397343744 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚76016943400960 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3805710756352 IDeviceStatsMonitor.on_train_batch_starÑG™æ" t/active_bytes.all.freed‚79822485157888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15397001 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚80546198565888 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2699 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54534949 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36142615 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3805541756928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682240 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚76106624301161 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚72312065033216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037ÛÁéÕ£2“- _wandbruntime‚2002  _runtime‚2002N…ØÀÐ&”ÊM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90105172 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35550438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54574799 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚76044212765696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15402494 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚79851334676480 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚76044212765696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3807103907840 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _step‚220 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3806934895104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3873603322880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _runtime‚2002.868325669 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90105172 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚80575135454208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20752483 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚76133956648041 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76044230768640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35550436 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36154975 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3796130593457 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 $ _timestamp‚1.788409392455488e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3795965471849 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54574799 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54554736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414720 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76134138809009 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35550436 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚79851147660800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90125237 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217w²ë`Õ&•ÏM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54574799 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚76044212765696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15402494 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚79851334676480 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚76044212765696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3807103907840 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _step‚220 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3806934895104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3873603322880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _runtime‚2002.868325669 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90105172 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚80575135454208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20752483 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚76133956648041 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76044230768640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35550436 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36154975 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3796130593457 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 $ _timestamp‚1.788409392455488e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3795965471849 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54574799 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54554736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414720 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76134138809009 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35550436 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚79851147660800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90125237 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90105172 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35550438 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2Ü]p~|2–- _wandbruntime‚2002  _runtime‚2002y8W#S—Í /loss/FRt4/detection_loss‚2.3584393918514253 1loss/SRqqt3/detection_loss‚1.6793017133393071 -loss/FBt1/detection_loss‚4.02392699321111 .loss/FRt2/detection_loss‚2.933233124514421 /loss/FRt1/assignment_loss‚5.006020630559614 2loss/SRqqt3/assignment_loss‚1.7119607886468822 0loss/FBt1/assignment_loss‚0.4358965416635935 2loss/SRqqt4/assignment_loss‚1.8252772753211586 0loss/FBt2/assignment_loss‚0.7194548164095197 .loss/FRt3/detection_loss‚2.082282863202549 .loss/FBt3/detection_loss‚1.063297545438611 0loss/FBt3/assignment_loss‚0.2219510104036999 1loss/SRqqt1/detection_loss‚2.9737296799818673 /loss/FBt4/detection_loss‚2.8544823492274567 0loss/SRqqt2/detection_loss‚1.643538223952055 /loss/FBt2/detection_loss‚2.5784287793295726 epoch‚2 .loss/FRt1/detection_loss‚3.133484971138739 0loss/FRt2/assignment_loss‚3.4686675336625843 /loss/FRt4/assignment_loss‚3.849560922384262 1loss/SRqqt2/assignment_loss‚1.647150618476527 /loss/FRt3/assignment_loss‚2.195014631464368 0loss/FBt4/assignment_loss‚0.5173826967035412 &loss/total_loss‚53.914889425338096 trainer/global_step‚2699 % _timestamp‚1.7884093924572666e+09 _step‚221 1loss/SRqqt1/assignment_loss‚2.892519657810529 0loss/SRqqt4/detection_loss‚2.099886666644703 _runtime‚2002.869321175RX˜Ò 0loss/FBt3/assignment_loss‚0.2219510104036999 1loss/SRqqt1/detection_loss‚2.9737296799818673 /loss/FBt4/detection_loss‚2.8544823492274567 0loss/SRqqt2/detection_loss‚1.643538223952055 /loss/FBt2/detection_loss‚2.5784287793295726 epoch‚2 .loss/FRt1/detection_loss‚3.133484971138739 0loss/FRt2/assignment_loss‚3.4686675336625843 /loss/FRt4/assignment_loss‚3.849560922384262 1loss/SRqqt2/assignment_loss‚1.647150618476527 /loss/FRt3/assignment_loss‚2.195014631464368 0loss/FBt4/assignment_loss‚0.5173826967035412 &loss/total_loss‚53.914889425338096 trainer/global_step‚2699 % _timestamp‚1.7884093924572666e+09 _step‚221 1loss/SRqqt1/assignment_loss‚2.892519657810529 0loss/SRqqt4/detection_loss‚2.099886666644703 _runtime‚2002.869321175 /loss/FRt4/detection_loss‚2.3584393918514253 1loss/SRqqt3/detection_loss‚1.6793017133393071 -loss/FBt1/detection_loss‚4.02392699321111 .loss/FRt2/detection_loss‚2.933233124514421 /loss/FRt1/assignment_loss‚5.006020630559614 2loss/SRqqt3/assignment_loss‚1.7119607886468822 0loss/FBt1/assignment_loss‚0.4358965416635935 2loss/SRqqt4/assignment_loss‚1.8252772753211586 0loss/FBt2/assignment_loss‚0.7194548164095197 .loss/FRt3/detection_loss‚2.082282863202549 .loss/FBt3/detection_loss‚1.063297545438611Ý7©°œ™‚(:‘ ·ìãÔò㦙 gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.834' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.966800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ñ:ßš‚(: ·ìãÔÉŽÜ›! disk.nvme0c0n1.in‚ 708.1796875( proc.memory.availableMB‚ 128.77734375 disk.rbd1.out‚ 1.2890625 disk.dm-0.in‚ 708.1796875 disk.sda2.in‚0 network.recv‚770719 disk.drbd1777.in‚0 disk.sda2.out‚0$ memory_percent‚99.21400547027588 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1777.out‚0$ disk.nvme0n1.out‚9023.7529296875 disk.dm-1.out‚ 86.2734375 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0% disk./.usageGB‚1056.2106857299805 disk.rbd2.out‚ 47.4736328125" proc.memory.rssMB‚ 2895.2578125 cpu‚25.36843485537718 proc.cpu.threads‚17 disk.rbd1.in‚0! disk.dm-0.out‚9023.7529296875 disk.sda3.out‚ 86.2734375) proc.memory.percent‚17.67125129699707 network.sent‚ 5.376454e+06) disk./.usagePercent‚70.88318533913257 disk.sda.out‚ 86.2734375 disk.sda1.in‚0& disk.nvme0c0n1.out‚9023.7529296875 disk.nvme0n1.in‚ 708.1796875 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0ÑÙ‘›‚(:† ÆìãÔŸô±™ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.532' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1064 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃúÃœ‚(:Œ ÆìãÔÉþÕœ network.recv‚777733 disk.rbd1.out‚ 1.2890625 disk.rbd2.out‚ 47.4736328125( proc.memory.availableMB‚ 128.76953125% disk./.usageGB‚1056.2106666564941 disk.rbd2.in‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 708.1796875 disk.sda2.out‚0) proc.memory.percent‚17.67125129699707 disk.rbd0.in‚0 disk.sda2.in‚0% disk.nvme0n1.out‚9024.30322265625 disk.dm-0.in‚ 708.1796875" disk.dm-0.out‚9024.30322265625 disk.rbd0.out‚0 disk.rbd3.in‚0' disk.nvme0c0n1.out‚9024.30322265625 disk.sda1.out‚0( disk./.usagePercent‚70.8831840590949 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 708.1796875# memory_percent‚99.2140531539917" proc.memory.rssMB‚ 2895.2578125 cpu‚25.366080196198045 network.sent‚ 5.485198e+06 disk.rbd3.out‚0 disk.sda3.out‚ 86.859375 disk.sda.out‚ 86.859375 disk.dm-1.in‚0 disk.dm-1.out‚ 86.859375 proc.cpu.threads‚17‡ŸÓœ2- _wandbruntime‚2037  _runtime‚2037ÿµ@–ÇžÁ trainer/global_step‚2749 % _timestamp‚1.7884094268844028e+09 _runtime‚2037.296496818 _step‚222 & lr-AdamW/pg1‚9.612289589630912e-05 & lr-AdamW/pg2‚9.612289589630912e-05èi‰ÕÌŸÆ % _timestamp‚1.7884094268844028e+09 _runtime‚2037.296496818 _step‚222 & lr-AdamW/pg1‚9.612289589630912e-05 & lr-AdamW/pg2‚9.612289589630912e-05 trainer/global_step‚2749Þà .2 - _wandbruntime‚2037  _runtime‚2037ê^X’g¡ÀO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3875213079552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚2749 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91698378 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36783709 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36170729 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55527651 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15669753 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚81255811719680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚77380411637760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚81255811719680 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81991869917184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55547688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3875213079552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238719488 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55547688 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚ 2037.29738425 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚77473437397129 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStaŽ/ì_tsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3875382078976 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36170727 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚81255624717312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36170729 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3943264711168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 % _timestamp‚1.7884094268850725e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚77380411637760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3943049931776 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _step‚223 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚81991845977088 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚77473255236369 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36170727 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚73609224150016 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3864048125713 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚85934895908864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3875382078976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21113956 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚85935134628352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3864213247113 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚73609207110656 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚77380429640704 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36783174 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55527651 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚77380429640704 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚81255624717312 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91718417 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1870023686,5 Ë'¢ÅO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21113956 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚85935134628352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3864213247113 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚73609207110656 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚77380429640704 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36783174 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55527651 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚77380429640704 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚81255624717312 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91718417 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3875213079552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚2749 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91698378 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36783709 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36170729 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55527651 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15669753 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚81255811719680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚77380411637760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚81255811719680 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81991869917184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55547688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3875213079552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238719488 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55547688 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚ 2037.29738425 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚77473437397129 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3875382078976 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36170727 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚81255624717312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36170729 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3943264711168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 % _timestamp‚1.7884094268850725e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚77380411637760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3943049931776 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _step‚223 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚81991845977088 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚77473255236369 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36170727 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚73609224150016 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3864048125713 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚85934895908864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3875382078976 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416ß·~·Ú2£- _wandbruntime‚2037  _runtime‚20374öÆJÐ&¤ÊM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204280320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _runtime‚2037.958261416 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3876606217728 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55567501 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚499 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228220416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3876775230464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36795921 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚77407681002496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚77407699005440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚81284287220224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3865437290257 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91750889 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚81284474235904 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚497 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3876775230464 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21121331 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚73635167332352 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚85965232988160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91730824 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚73635150292992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 trainer/global_step‚2749 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77500769744217 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36183386 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91750889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36183388 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3944678343168 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36183386 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82020782865408 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3944474062848 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36796420 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55567501 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55547438 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82020782865408 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3944474062848 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36796420 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55567501 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55547438 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3876775230464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36795921 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚77407681002496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚77407699005440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚81284287220224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3865437290257 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91750889 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚81284474235904 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚497 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3876775230464 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21121331 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚73635167332352 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚85965232988160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ôš©e$ 91730824 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚73635150292992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 trainer/global_step‚2749 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77500769744217 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36183386 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91750889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36183388 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3944678343168 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36183386 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3944884402176 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚74906366227456 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚2799 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 _runtime‚2072.637625096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37425799 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93324030 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37425302 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87451473748992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚78743897877504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78743879874560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3945053401600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚78839886171577 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82688764276736 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3945053401600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21482806 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚74906383266816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3944884402176 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚74906366227456 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚2799 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 _runtime‚2072.637625096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37425799 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93324030 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37425302 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87451473748992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚78743897877504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78743879874560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3945053401600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚78839886171577 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82688764276736ãÝ¢/è2µ- _wandbruntime‚2073  _runtime‚2073zx³¶ËM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93356476 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚2073.297863074 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56560203 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚78771167242240 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚87481798538752 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83466430276608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3934909108665 RDDÕ”¡"eviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3946277540352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚474 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4015392202240 % _timestamp‚1.7884094628852994e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36816338 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36816336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15947691 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3946277540352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚87482039342080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚476 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚74932309409792 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93376541 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚82717426779648 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4015609065472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93376541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21490181 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83466406336512 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216863232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚78771149239296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37438348 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚78867218518457 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3935074230273 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37437872 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15948165 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2799 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93356476 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚78771149239296 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚78771167242240 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚82717613795328 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚74932326449152 _step‚228 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21490183 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36816336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56540140 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56540140ÜÄAŸÖ&·ÐM QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3946277540352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚87482039342080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚476 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚74932309409792 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93376541 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚82717426779648 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4015609065472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93376541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21490181 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83466406336512 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216863232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚78771149239296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37438348 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚78867218518457 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3935074230273 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37437872 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15948165 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2799 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93356476 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚78771149239296 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚78771167242240 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚82717613795328 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚74932326449152 _step‚228 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21490183 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36816336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56540140 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56540140 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93356476 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚2073.297863074 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56560203 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚78771167242240 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚87481798538752 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83466430276608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3934909108665 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3946277540352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚474 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4015392202240 % _timestamp‚1.7884094628852994e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36816338 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36816336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15947691 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360ä_Vc–2¸- _wandbruntime‚2073  _runtime‚2073€œwUW¹Ñ /loss/FRt4/assignment_loss‚4.158685690412919 1loss/FBt2/assignment_loss‚0.30704266572229244 /loss/FRt4/detection_loss‚2.2252905406057835 1loss/SRqqt2/detection_loss‚1.7183272260885971 1loss/SRqqt3/detection_loss‚1.9913306014405356 _step‚229 /loss/FRt2/detection_loss‚2.1344930069488393 trainer/global_step‚2799 1loss/FBt1/assignment_loss‚0.24610914760738856 0loss/SRqqt1/detection_loss‚2.888023316860199 0loss/FRt3/assignment_loss‚3.1358493122648685 0loss/FBt3/assignment_loss‚0.4513517791311712 0loss/FBt4/assignment_loss‚0.8221322704650144 /loss/FRt3/detection_loss‚2.2906178346973785 _runtime‚2073.299328326 .loss/FRt1/assignment_loss‚4.80562500726609 /loss/FRt2/assignment_loss‚3.117841833516171 1loss/SRqqt4/detection_loss‚2.0321853775531054 .loss/FBt2/detection_loss‚2.227719962601036 %loss/total_loss‚55.02918509207473 2loss/SRqqt2/assignment_loss‚1.5783693091227458 2loss/SRqqt3/assignment_loss‚2.0402760512299007 .loss/FBt3/detection_loss‚1.031267855358323 % _timestamp‚1.7884094628872414e+09 2loss/SRqqt1/assignment_loss‚2.0882102225137795 1loss/SRqqt4/assignment_loss‚2.122763198489944 /loss/FRt1/detection_loss‚3.4858401389349076 epoch‚2 .loss/FBt1/detection_loss‚4.801200075583025 .loss/FBt4/detection_loss‚3.328632667660713D(X9\ºÖ /loss/FRt2/detection_loss‚2.1344930069488393 trainer/global_step‚2799 1loss/FBt1/assignment_loss‚0.24610914760738856 0loss/SRqqt1/detection_loss‚2.888023316860199 0loss/FRt3/assignment_loss‚3.1358493122648685 0loss/FBt3/assignment_loss‚0.4513517791311712 0loss/FBt4/assignment_loss‚0.8221322704650144 /loss/FRt3/detection_loss‚2.2906178346973785 _runtime‚2073.299328326 .loss/FRt1/assignment_loss‚4.80562500726609 /loss/FRt2/assignment_loss‚3.117841833516171 1loss/SRqqt4/detection_loss‚2.0321853775531054 .loss/FBt2/detection_loss‚2.227719962601036 %loss/total_loss‚55.02918509207473 2loss/SRqqt2/assignment_loss‚1.5783693091227458 2loss/SRqqt3/assignment_loss‚2.0402760512299007 .loss/FBt3/detection_loss‚1.031267855358323 % _timestamp‚1.7884094628872414e+09 2loss/SRqqt1/assignment_loss‚2.0882102225137795 1loss/SRqqt4/assignment_loss‚2.122763198489944 /loss/FRt1/detection_loss‚3.4858401389349076 epoch‚2 .loss/FBt1/detection_loss‚4.801200075583025 .loss/FBt4/detection_loss‚3.328632667660713 /loss/FRt4/assignment_loss‚4.158685690412919 1loss/FBt2/assignment_loss‚0.30704266572229244 /loss/FRt4/detection_loss‚2.2252905406057835 1loss/SRqqt2/detection_loss‚1.7183272260885971 1loss/SRqqt3/detection_loss‚1.9913306014405356 _step‚229åé௜»‚(:‘ ‚íãÔÍ´Æ™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.038' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.407600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û*ÿ¼‚(:‘ ‚íãÔ¿ÛÚœ disk.sda.out‚ 89.23828125' proc.memory.availableMB‚ 128.3671875$ disk./.usageGB‚1056.210952758789! disk.nvme0c0n1.in‚ 708.1796875 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.rbd2.out‚ 49.236328125 disk.rbd3.in‚0 network.sent‚ 5.610892e+06% disk.nvme0n1.out‚9026.92236328125" disk.dm-0.out‚9026.92236328125" proc.memory.rssMB‚ 2895.2578125 cpu‚25.357089587513418 proc.cpu.threads‚17 disk.dm-0.in‚ 708.1796875 network.recv‚797017 disk.sda2.in‚0 disk.nvme0n1.in‚ 708.1796875 disk.rbd0.in‚0 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.2890625 disk.dm-1.out‚ 89.23828125 disk.rbd0.out‚0 disk.rbd2.in‚0$ memory_percent‚99.21650886535645) disk./.usagePercent‚70.88320325965996' disk.nvme0c0n1.out‚9026.92236328125 disk.sda2.out‚0 disk.sda3.out‚ 89.23828125 disk.dm-1.in‚0) proc.memory.percent‚17.67125129699707 disk.rbd3.out‚0ökÚ›½‚(: ‘íãÔñµ¦™ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚96.309' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.261799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b®¢M¾‚(:Œ ‘íãÔ¬¦Ë› disk.rbd1.in‚0 disk.dm-1.in‚0 disk.rbd2.in‚0 network.recv‚803714 disk.sda2.in‚0 disk.rbd1.out‚ 1.2890625% disk.nvme0n1.out‚9028.94189453125" disk.dm-0.out‚9028.94189453125 disk.rbd0.out‚0 cpu‚25.35483142327739) disk./.usagePercent‚70.87188363052599 disk.sda2.out‚0 network.sent‚ 5.719519e+06 disk.sda3.in‚0 disk.sda.out‚ 89.9765625 disk.nvme0n1.in‚ 708.1796875 disk.dm-1.out‚ 89.9765625" proc.memory.rssMB‚ 2895.2578125 proc.cpu.threads‚17 disk.sda3.out‚ 89.9765625 disk.sda.in‚0 disk.sda1.in‚0% disk./.usageGB‚1056.0422821044922 disk.rbd3.in‚0' disk.nvme0c0n1.out‚9028.94189453125$ memory_percent‚99.21646118164062) proc.memory.percent‚17.67125129699707 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd2.out‚49.40087890625# proc.memory.availableMB‚128.375 disk.rbd3.out‚0! disk.nvme0c0n1.in‚ 708.1796875 disk.dm-0.in‚ 708.1796875 disk.rbd0.in‚0Œ¦Åõ2¿- _wandbruntime‚2107  _runtime‚2107Gã©ÆÀÀ trainer/global_step‚2849 % _timestamp‚1.7884094970320399e+09 _runtime‚ 2107.44422286 _step‚230 & lr-AdamW/pg1‚9.547661363928235e-05 & lr-AdamW/pg2‚9.547661363928235e-05øðÜ'ËÁÅ & lr-AdamW/pg1‚9.547661363928235e-05 & lr-AdamW/pg2‚9.547661363928235e-05 trainer/global_step‚2849 % _timestamp‚1.7884094970320399e+09 _runtime‚ 2107.44422286 _step‚230æÿKTö2Â- _wandbruntime‚2107  _runtime‚2107 ~ÎfÃÁO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21851654 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4014724724224 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57533092 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37436627 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚84883164739584 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4002991762529 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38063779 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16212125 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232428032 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21851656 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚88968246010880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4003156883929 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚84122090838528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84121903836160 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚80107348111360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚76203542383616 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37436627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57513055 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚80107348111360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4014724724224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80206517106785 XJDeviceStatsMonitor.on_train_batch_stùu3a art/allocated_bytes.small_pool.current‚ 168999424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚76203525344256 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38064323 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚84883140799488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚88968013582848 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚80206699267545 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4085081271296 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚231 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚80107366114304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94949682 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4084872783360Ÿ}ìÌ'ÄÆO ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚88968246010880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4003156883929 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚84122090838528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84121903836160 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚80107348111360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚76203542383616 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37436627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57513055 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚80107348111360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4014724724224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80206517106785 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚76203525344256 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38064323 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚84883140799488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚88968013582848 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚80206699267545 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4085081271296 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚231 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚80107366114304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94949682 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4084872783360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21851654 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4014724724224 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57533092 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37436627 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚84883164739584 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4002991762529 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38063779 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16212125 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232428032 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21851656 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2ç´ç2Å- _wandbruntime‚2108  _runtime‚2108ålŽŠÑ&ÆËM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37449288 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4016117875712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38076999 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21859033 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80233849453665 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4004546048681 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 % _timestamp‚1.7884094976814601e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4016117875712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4004380927073 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95002193 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57552905 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚80134617476096 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚88998344436736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76229468526592 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚80134635479040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚84912077687808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚2108.094134045 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37449288 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚80134617476096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38076472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16217966 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 trainer/global_step‚2849 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57552905 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57532842 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚80134635479040 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37449286 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94982128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84150753354752 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95002193 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4086290689024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4015948862976 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84150753354752 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚84912053747712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 _step‚232 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37449286 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4015948862976lt?Ö&ÇÐM ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95002193 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57552905 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚80134617476096 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚88998344436736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76229468526592 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚80134635479040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚84912077687808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚2108.094134045 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37449288 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚80134617476096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38076472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16217966 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 trainer/global_step‚2849 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57552905 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57532842 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚80134635479040 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37449286 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94982128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84150753354752 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95002193 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4086290689024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4015948862976 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84150753354752 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚84912053747712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 _step‚232 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37449286 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4015948862976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37449288 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4016117875712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38076999 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21859033 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80233849453665 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4004546048681 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 % _timestamp‚1.7884094976814601e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4016117875712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4004380927073è6R~2È- _wandbruntime‚2108  _runtime‚2108û e—ÉÏ /loss/FRt1/assignment_loss‚4.736915074288845 2loss/SRqqt1/assignment_loss‚2.2761124637391834 2loss/SRqqt4/assignment_loss‚2.02992710íEaì¾36122666 1loss/SRqqt4/detection_loss‚1.6643117847012692 _runtime‚2108.095530337 .loss/FRt4/detection_loss‚2.076870108453127 1loss/SRqqt3/detection_loss‚2.0146519062774524 /loss/FBt3/detection_loss‚1.1592783196856493 0loss/FRt3/assignment_loss‚2.3550740987062455 1loss/SRqqt2/assignment_loss‚1.828669034750075 2loss/SRqqt3/assignment_loss‚2.0974725775775456 0loss/FBt1/assignment_loss‚0.4581803789224068 /loss/FRt4/assignment_loss‚3.831332973562754 1loss/FBt3/assignment_loss‚0.15402506002145344 0loss/SRqqt1/detection_loss‚3.456893563270569 /loss/FBt4/detection_loss‚3.2012641727924347 epoch‚2 trainer/global_step‚2849 _step‚233 0loss/FRt2/assignment_loss‚3.3661982119083405 % _timestamp‚1.7884094976834583e+09 1loss/SRqqt2/detection_loss‚1.8559621567963749 1loss/FBt2/assignment_loss‚0.45847908575087787 1loss/FBt4/assignment_loss‚0.43296698869119155 &loss/total_loss‚55.323300301201115 *loss/FRt1/detection_loss‚ 2.75672634691 /loss/FRt2/detection_loss‚2.3574844103838717 /loss/FRt3/detection_loss‚2.2626195695251226 .loss/FBt1/detection_loss‚5.660400249063969 ,loss/FBt2/detection_loss‚2.8314846618101îf» ZÊÔ 1loss/SRqqt2/detection_loss‚1.8559621567963749 1loss/FBt2/assignment_loss‚0.45847908575087787 1loss/FBt4/assignment_loss‚0.43296698869119155 &loss/total_loss‚55.323300301201115 *loss/FRt1/detection_loss‚ 2.75672634691 /loss/FRt2/detection_loss‚2.3574844103838717 /loss/FRt3/detection_loss‚2.2626195695251226 .loss/FBt1/detection_loss‚5.660400249063969 ,loss/FBt2/detection_loss‚2.8314846618101 /loss/FRt1/assignment_loss‚4.736915074288845 2loss/SRqqt1/assignment_loss‚2.2761124637391834 2loss/SRqqt4/assignment_loss‚2.0299271036122666 1loss/SRqqt4/detection_loss‚1.6643117847012692 _runtime‚2108.095530337 .loss/FRt4/detection_loss‚2.076870108453127 1loss/SRqqt3/detection_loss‚2.0146519062774524 /loss/FBt3/detection_loss‚1.1592783196856493 0loss/FRt3/assignment_loss‚2.3550740987062455 1loss/SRqqt2/assignment_loss‚1.828669034750075 2loss/SRqqt3/assignment_loss‚2.0974725775775456 0loss/FBt1/assignment_loss‚0.4581803789224068 /loss/FRt4/assignment_loss‚3.831332973562754 1loss/FBt3/assignment_loss‚0.15402506002145344 0loss/SRqqt1/detection_loss‚3.456893563270569 /loss/FBt4/detection_loss‚3.2012641727924347 epoch‚2 trainer/global_step‚2849 _step‚233 0loss/FRt2/assignment_loss‚3.3661982119083405 % _timestamp‚1.7884094976834583e+09é.J/œË‚(:‘  íãÔž·‡™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.821' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.164199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·¤ø^Ì‚(:”  íãÔøí»›$ disk./.usageGB‚1056.040615081787 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda.in‚0$ memory_percent‚99.21658039093018 cpu‚25.352580644071203 disk.sda.out‚ 90.33203125" disk.dm-0.out‚9030.20556640625 disk.dm-1.in‚0( proc.memory.availableMB‚ 128.35546875 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚49.40087890625' disk.nvme0c0n1.out‚9030.20556640625" proc.memory.rssMB‚ 2895.2578125 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.dm-1.out‚ 90.33203125 disk.rbd3.in‚0 network.recv‚807727) disk./.usagePercent‚70.87177175523352 disk.sda3.out‚ 90.33203125 disk.nvme0n1.in‚ 708.1796875 disk.dm-0.in‚ 708.1796875 disk.rbd0.out‚0 disk.sda3.in‚0% disk.nvme0n1.out‚9030.20556640625! disk.nvme0c0n1.in‚ 708.1796875) proc.memory.percent‚17.67125129699707 network.sent‚ 5.724877e+06 disk.rbd1.out‚ 1.2890625 disk.rbd2.in‚0 disk.rbd3.out‚0t®‘Í‚(:† ¯íãÔ€Ö¡™ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.454' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8908 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸ ΂(:• ¯íãÔ¦²Ë›! disk.nvme0c0n1.in‚ 708.1796875 disk.rbd0.in‚0 disk.sda.out‚ 91.28515625 disk.dm-1.out‚ 91.28515625 disk.rbd3.out‚0' disk.nvme0c0n1.out‚9030.63818359375 disk.sda3.out‚ 91.28515625 disk.nvme0n1.in‚ 708.1796875 disk.rbd2.out‚49.47900390625 disk.sda2.in‚0" proc.memory.rssMB‚ 2895.2578125 network.sent‚ 5.832887e+06 disk.drbd1777.out‚0 cpu‚25.350471105171028 proc.cpu.threads‚17% disk./.usageGB‚1056.0408325195312 disk.sda1.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚9030.63818359375 disk.dm-1.in‚0 disk.rbd2.in‚0) proc.memory.percent‚17.67125129699707 network.recv‚813708 disk.drbd1777.in‚0 disk.rbd1.out‚ 1.2890625 disk.sda2.out‚0 disk.sda3.in‚0) disk./.usagePercent‚70.87178634766298 disk.sda1.out‚0" disk.dm-0.out‚9030.63818359375 disk.rbd0.out‚0 disk.rbd3.in‚0$ memory_percent‚99.21553134918213( proc.memory.availableMB‚ 128.52734375 disk.sda.in‚0 disk.dm-0.in‚ 708.1796875RZm¥2Ï- _wandbruntime‚2142  _runtime‚2142$“:ÇÐÁ trainer/global_step‚2899 % _timestamp‚1.7884095318211868e+09 _runtime‚2142.233361002 _step‚234 & lr-AdamW/pg1‚9.513938419500803e-05 & lr-AdamW/pg2‚9.513938419500803e-05¨{µÂÌÑÆ & lr-AdamW/pg2‚9.513938419500803e-05 trainer/global_step‚2899 % _timestamp‚1.7884095318211868e+09 _runtime‚2142.233361002 _step‚234 & lr-AdamW/pg1‚9.513938419500803e-05ê­ržC2Ò- _wandbruntime‚2142  _runtime‚2142UFn‡Æ'ÓÀO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4072628702337 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58525794 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58505757 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4155924454400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4155718063616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚85555043395584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230330880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚81470816348160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚85555230397952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚77500701500416 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 _runtime‚ 2142.23430991 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16483964 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚86328812150784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚81470816348160 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96575334 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4084396046848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85555043395584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚81470834351104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚77500684461056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚90484736605184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38069577 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38704468 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38704964 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4084227047424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16484458 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚90484506274304 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚86328788210688 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _step‚235 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58525794 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚85555230397952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22220506 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38069579 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0ìP\SË'ÔÅO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96575334 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4084396046848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85555043395584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚81470834351104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚77500684461056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚90484736605184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38069577 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38704468 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38704964 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4084227047424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16484458 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚90484506274304 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚86328788210688 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _step‚235 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58525794 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚85555230397952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22220506 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38069579 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4072628702337 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58525794 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58505757 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4155924454400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4155718063616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚85555043395584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230330880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚81470816348160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚85555230397952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚77500701500416 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 _runtime‚ 2142.23430991 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16483964 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚86328812150784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚81470816348160 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368ëV³ôY2Õ- _wandbruntime‚2142  _runtime‚2142 ³á`ÖËM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4085620185600 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚77526627643392 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96627845 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4085620185600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 _runtime‚2142.885559613 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16489857 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚90515070243328 % _timestamp‚1.7884095324728131e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4085789198336 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96627845 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚90514835731456 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22227883 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚81600662549841 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81498085712896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4085789198336 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38717740 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚85583892914176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571776 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚508 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4157134572544 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58545607 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38717232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚77526644682752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4074017867089 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 MCDeviceSta[²‡£qtsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚236 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38082238 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16489351 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚81498085712896 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚85583892914176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚86357725099008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚81498103715840 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22227881 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚81498085712896 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚85583892914176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚86357725099008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚81498103715840 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22227881 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚77526627643392 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96627845 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4085620185600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 _runtime‚2142.885559613 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16489857 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚90515070243328 % _timestamp‚1.7884095324728131e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4085789198336 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96627845 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚90514835731456 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22227883 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚81600662549841 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81498085712896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4085789198336 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38717740 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚85583892914176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571776 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚508 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4157134572544 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58545607 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38717232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚77526644682752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4074017867089 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚236ìÔG¸'2Ø- _wandbruntime‚2142  _runtime‚2142@ÜUVÙÐ 0loss/FRt4/assignment_loss‚4.3859157156437005 1loss/SRqqt1/detection_loss‚2.7986783775789985 .loss/FBt4/detection_loss‚2.821061532696088 trainer/global_step‚2899 2loss/SRqqt2/assignment_loss‚1.7779800860211252 1loss/FBt3/assignment_loss‚0.36745021182706783 /loss/FBt4/assignment_loss‚0.699120157689625 /loss/FRt2/assignment_loss‚3.777077944911256 2loss/SRqqt4/assignment_loss‚2.2607420290941778 epoch‚2 $ _timestamp‚1.788409532474509e+09 /loss/FRt1/assignment_loss‚4.874476112817463 .loss/FRt3/detection_loss‚2.373882283270359 .loss/FRt4/detection_loss‚2.068030441060979 &loss/total_loss‚56.042722615069295 1loss/SRqqt3/assignment_loss‚2.096629098653793 /loss/FRt2/detection_loss‚2.2906995038596953 1loss/SRqqt3/detection_loss‚1.9251243644952774 /loss/FBt3/detection_loss‚0.9943877731659309 _step‚237 /loss/FRt1/detection_loss‚2.6837559201215444 0loss/SRqqt2/detection_loss‚1.455790501460433 .loss/FBt1/detection_loss‚4.821893082724677 2loss/SRqqt1/assignment_loss‚2.7175526362040947 0loss/FBt1/assignment_loss‚0.6717914136226883 0loss/FBt2/assignment_loss‚0.6881364456244877 .loss/FBt2/detection_loss‚2.022913984416976 0loss/FRt3/assignment_loss‚3.3059058487415314 1loss/SRqqt4/detection_loss‚2.1637271493673325 _runtime‚2142.886562353ŒhLf[ÚÕ 0loss/FRt3/assignment_loss‚3.3059058487415314 1loss/SRqqt4/detection_loss‚2.1637271493673325 _runtime‚2142.886562353 0loss/FRt4/assignment_loss‚4.3859157156437005 1loss/SRqqt1/detection_loss‚2.7986783775789985 .loss/FBt4/detection_loss‚2.821061532696088 trainer/global_step‚2899 2loss/SRqqt2/assignment_loss‚1.7779800860211252 1loss/FBt3/assignment_loss‚0.36745021182706783 /loss/FBt4/assignment_loss‚0.699120157689625 /loss/FRt2/assignment_loss‚3.777077944911256 2loss/SRqqt4/assignment_loss‚2.2607420290941778 epoch‚2 $ _timestamp‚1.788409532474509e+09 /loss/FRt1/assignment_loss‚4.874476112817463 .loss/FRt3/detection_loss‚2.373882283270359 .loss/FRt4/detection_loss‚2.068030441060979 &loss/total_loss‚56.042722615069295 1loss/SRqqt3/assignment_loss‚2.096629098653793 /loss/FRt2/detection_loss‚2.2906995038596953 1loss/SRqqt3/detection_loss‚1.9251243644952774 /loss/FBt3/detection_loss‚0.9943877731659309 _step‚237 /loss/FRt1/detection_loss‚2.6837559201215444 0loss/SRqqt2/detection_loss‚1.455790501460433 .loss/FBt1/detection_loss‚4.821893082724677 2loss/SRqqt1/assignment_loss‚2.7175526362040947 0loss/FBt1/assignment_loss‚0.6717914136226883 0loss/FBt2/assignment_loss‚0.6881364456244877 .loss/FBt2/detection_loss‚2.022913984416976íÊõ–œÛ‚(:‘ ¾íãÔ‚ž·™ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.834' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.966800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?ú¸ Ü‚(:“ ¾íãÔ¬¼ÿ› disk.rbd0.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 708.1796875% disk./.usageGB‚1056.1708755493164 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda.out‚ 91.91015625 disk.sda1.in‚0 cpu‚25.34826854183064 disk.dm-0.in‚ 708.1796875 disk.sda1.out‚0 network.sent‚ 5.839132e+06 network.recv‚818763) disk./.usagePercent‚70.88051364450524" disk.dm-0.out‚9164.51708984375 disk.drbd1777.out‚0" proc.memory.rssMB‚ 2895.2578125 disk.sda2.out‚0 disk.sda3.in‚0) proc.memory.percent‚17.67125129699707 disk.rbd1.in‚0 disk.rbd1.out‚ 1.2890625 disk.dm-1.out‚ 91.91015625 disk.rbd2.out‚51.05322265625' disk.nvme0c0n1.out‚9164.51708984375 disk.nvme0n1.in‚ 708.1796875% disk.nvme0n1.out‚9164.51708984375 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 91.91015625 disk.drbd1777.in‚0$ memory_percent‚99.21727180480957' proc.memory.availableMB‚ 128.2421875Ç¡ÌXÝ‚(:… ÍíãÔåñ«™ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.864' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5728 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•¼Û;Þ‚(:Š ÍíãÔ­¯§œ disk.sda3.in‚0 disk.rbd1.in‚0! disk.dm-0.out‚9304.2783203125 disk.rbd2.in‚0 network.recv‚825124 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 710.09375& disk.nvme0c0n1.out‚9304.2783203125" proc.memory.rssMB‚ 2895.2578125 disk.sda.in‚0 disk.sda.out‚ 93.80859375$ disk.nvme0n1.out‚9304.2783203125& proc.memory.availableMB‚ 128.265625 cpu‚25.346213137217873) disk./.usagePercent‚70.83312972200777 disk.rbd1.out‚ 1.2890625) proc.memory.percent‚17.67125129699707 network.sent‚ 5.947837e+06 disk.sda2.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 710.09375 disk.sda3.out‚ 93.80859375 disk.sda1.out‚0 disk.nvme0n1.in‚ 710.09375 disk.dm-1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1055.4648208618164 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.dm-1.out‚ 93.80859375 disk.rbd2.out‚51.19384765625$ memory_percent‚99.21712875366211$=Šmß‚(:… ÜíãÔÔóš™ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚29.0 gpu.0.powerWatts‚80.249' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚16.0498 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(„ëà‚(:‰ ÜíãÔù”œ network.sent‚ 5.953155e+06 disk.sda2.out‚0 disk.sda3.out‚ 94.4453125 disk.rbd1.out‚ 1.2890625! disk.dm-0.out‚9316.6533203125) proc.memory.percent‚17.67125129699707 disk.dm-1.out‚ 94.4453125 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda.out‚ 94.4453125% disk./.usageGB‚1055.4647216796875 disk.rbd0.in‚0& disk.nvme0c0n1.out‚9316.6533203125 disk.drbd1777.out‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚9316.6533203125 disk.rbd2.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 128.24609375 proc.cpu.threads‚17) disk./.usagePercent‚70.83312306581188 disk.rbd3.in‚0 disk.sda3.in‚0 disk.dm-0.in‚ 710.09375 disk.rbd2.out‚51.19384765625 disk.nvme0c0n1.in‚ 710.09375$ memory_percent‚99.21724796295166 network.recv‚828957 disk.dm-1.in‚0 disk.nvme0n1.in‚ 710.09375 cpu‚25.344195158260558 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0" proc.memory.rssMB‚ 2895.2578125Ïô%Ž2á- _wandbruntime‚2179  _runtime‚2179‚‰±€ÇâÁ % _timestamp‚1.7884095688916278e+09 _runtime‚2179.303779819 _step‚238 & lr-AdamW/pg1‚9.479285559351532e-05 & lr-AdamW/pg2‚9.479285559351532e-05 trainer/global_step‚2949Dc÷ÌãÆ _runtime‚2179.303779819 _step‚238 & lr-AdamW/pg1‚9.479285559351532e-05 & lr-AdamW/pg2‚9.479285559351532e-05 trainer/global_step‚2949 % _timestamp‚1.7884095688916278e+09îSmŽQ2ä- _wandbruntime‚2179  _runtime‚21795¸¸xÇ'åÁO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4226580238336 % _timestamp‚1.7884095688923554e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚82834284584960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4153898370048 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98221025 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22589354 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚92001015860224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚87774459561984 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38702529 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4153898370048 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39344437 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59518496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚92001250385408 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38702527 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38702529 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4226790823424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _step‚239 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚86988369957376 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚86988182955008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4154067369472 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87774435621888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86988182955008 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚78797860617216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16755612 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚82939961137961 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚86988369957376 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚2949 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚2179.304684393 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59498459 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚78797843577856 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59498459 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4141935399345 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚82834302587904 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59518496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38702527[»OÞžæÆO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4153898370048 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39344437 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59518496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚92001250385408 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 JBDeviceStatsMonitor.on_train_batch_start/inactive¤6Ì™.#_split.all.current‚531 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38702527 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38702529 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4226790823424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _step‚239 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚86988369957376 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚86988182955008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4154067369472 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87774435621888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86988182955008 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚78797860617216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16755612 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚82939961137961 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚86988369957376 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚2949 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚2179.304684393 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59498459 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚78797843577856 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59498459 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4141935399345 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚82834302587904 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59518496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38702527 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4226580238336 % _timestamp‚1.7884095688923554e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚82834284584960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4153898370048 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98221025 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22589354 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚92001015860224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚87774459561984 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38702529 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0ï'2ç- _wandbruntime‚2179  _runtime‚2179r¿«Ñ&èËM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4143324563889 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59538309 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16760884 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98233432 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39357617 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22596733 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚82967111324081 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚87017032473600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16760364 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚87803372510208 trainer/global_step‚2949 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚82861553949696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38715188 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39357095 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4155460520960 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59518246 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4155291508224 _runtime‚2179.973759058 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚82861553949696 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚92031346439680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4227997869568 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚78823803799552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 % _timestamp‚1.7884095695610356e+09 _step‚240 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚87803348570112 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98253497 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4155291508224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59518246 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38715188 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59538309 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚87017032473600 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98233432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38715186 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚92031578854400 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚82967293485049RÏ/Ö&éÐM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59518246 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38715188 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59538309 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚87017032473600 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98233432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38715186 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚92031578854400 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚82967293485049 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4143324563889 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59538309 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16760884 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98233432 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39357617 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22596733 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚82967111324081 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚87017032473600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16760364 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚87803372510208 trainer/global_step‚2949 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚82861553949696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38715188 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39357095 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4155460520960 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59518246 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4155291508224 _runtime‚2179.973759058 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚82861553949696 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚92031346439680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4227997869568 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚78823803799552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 % _timestamp‚1.7884095695610356e+09 _step‚240 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚87803348570112 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98253497 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4155291508224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0ð¥î[e2ê- _wandbruntime‚2179  _runtime‚2179úã~|\ëÖ /loss/FRt1/assignment_loss‚4.912856294049157 /loss/FRt2/assignment_loss‚3.642161352130083 1loss/SRqqt2/assignment_loss‚1.705841020729981 0loss/FBt2/assignment_loss‚0.8287396756085482 /loss/FRt2/detection_loss‚2.1680055868167143 /loss/FBt1/detection_loss‚3.8565640321799686 1loss/FBt3/assignment_loss‚0.36741689041714976 /loss/FBt4/detection_loss‚2.7738535631270635 _runtime‚2179.975259946 1loss/SRqqt1/assignment_loss‚2.423075266679128 2loss/SRqqt3/assignment_loss‚2.3627152485044105 .loss/FRt3/detection_loss‚2.041689936603819 1loss/SRqqt1/detection_loss‚2.5904321124156318 1loss/SRqqt2/detection_loss‚1.4481282053809417 1loss/SRqqt3/detection_loss‚1.7796918851849826 % _timestamp‚1.7884095695631726e+09 0loss/FRt3/assignment_loss‚3.4413621644584502 1loss/SRqqt4/detection_loss‚1.8118204428599431 trainer/global_step‚2949 0loss/FBt4/assignment_loss‚0.3208278891337657 /loss/FBt3/detection_loss‚1.2506972144068849 2loss/SRqqt4/assignment_loss‚1.9091209061443806 /loss/FRt4/detection_loss‚2.1997192135701575 /loss/FBt2/detection_loss‚1.4311394454019428 &loss/total_loss‚52.399452759691684 /loss/FRt4/assignment_loss‚4.159403078258038 /loss/FRt1/detection_loss‚2.7486314906014337 epoch‚2 _step‚241 1loss/FBt1/assignment_loss‚0.22555984502910828#åØaìÛ epoch‚2 _step‚241 1loss/FBt1/assignment_loss‚0.22555984502910828 /loss/FRt1/assignment_loss‚4.912856294049157 /loss/FRt2/assignment_loss‚3.642161352130083 1loss/SRqqt2/assignment_loss‚1.705841020729981 0loss/FBt2/assignment_loss‚0.8287396756085482 /loss/FRt2/detection_loss‚2.1680055868167143 /loss/FBt1/detection_loss‚3.8565640321799686 1loss/FBt3/assignment_loss‚0.36741689041714976 /loss/FBt4/detection_loss‚2.7738535631270635 _runtime‚2179.975259946 1loss/SRqqt1/assignment_loss‚2.423075266679128 2loss/SRqqt3/assignment_loss‚2.3627152485044105 .loss/FRt3/detection_loss‚2.041689936603819 1loss/SRqqt1/detection_loss‚2.5904321124156318 1loss/SRqqt2/detection_loss‚1.4481282053809417 1loss/SRqqt3/detection_loss‚1.7796918851849826 % _timestamp‚1.7884095695631726e+09 0loss/FRt3/assignment_loss‚3.4413621644584502 1loss/SRqqt4/detection_loss‚1.8118204428599431 trainer/global_step‚2949 0loss/FBt4/assignment_loss‚0.3208278891337657 /loss/FBt3/detection_loss‚1.2506972144068849 2loss/SRqqt4/assignment_loss‚1.9091209061443806 /loss/FRt4/detection_loss‚2.1997192135701575 /loss/FBt2/detection_loss‚1.4311394454019428 &loss/total_loss‚52.399452759691684 /loss/FRt4/assignment_loss‚4.159403078258038 /loss/FRt1/detection_loss‚2.7486314906014337ñ:—¡‘í‚(:† ëíãԵ矙 gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.184' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.(f:î‚(: ëíãÔêãÈ› network.sent‚ 5.95883e+06 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 710.1015625 disk.sda3.in‚0) proc.memory.percent‚17.67125129699707 cpu‚25.342218068889093) disk./.usagePercent‚71.28590183098227% disk.nvme0n1.out‚13541.2724609375' disk.nvme0c0n1.out‚13541.2724609375 disk.sda2.in‚0% disk./.usageGB‚1062.2114524841309" disk.dm-0.out‚13541.2724609375 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda.out‚ 96.1484375 disk.sda1.out‚0 disk.rbd2.out‚ 51.2958984375  F>Þ disk.sda3.out‚ 96.1484375% proc.memory.availableMB‚ 128.46875" proc.memory.rssMB‚ 2895.2578125 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 710.1015625 disk.dm-0.in‚ 710.1015625 disk.dm-1.out‚ 96.1484375 disk.sda2.out‚0$ memory_percent‚99.21588897705078 network.recv‚832942 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.2890625 disk.rbd0.out‚0 disk.rbd2.in‚0ÉÊ¢ï‚(:… úíãÔ©ý£™ gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚89.717' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.9434 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y¾ Œð‚(:‘ úíãÔŽœÜ› proc.cpu.threads‚17 disk.sda1.out‚0& disk.nvme0n1.out‚17986.41357421875 disk.rbd0.out‚0 disk.sda3.out‚ 96.7890625 disk.sda2.in‚0 disk.sda.out‚ 96.7890625 network.sent‚ 6.067521e+06% disk./.usageGB‚1066.8322639465332 disk.rbd1.out‚ 1.2890625 disk.nvme0n1.in‚ 710.1015625 disk.rbd0.in‚0$ memory_percent‚99.21770095825195 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd2.out‚ 51.3037109375 disk.rbd2.in‚0! disk.nvme0c0n1.in‚ 710.1015625& proc.memory.availableMB‚ 128.171875" proc.memory.rssMB‚ 2895.2578125 cpu‚25.34024858824357 network.recv‚840092 disk.dm-1.in‚0 disk.dm-1.out‚ 96.7890625 disk.sda.in‚0 disk.sda1.in‚0) proc.memory.percent‚17.67125129699707 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0) disk./.usagePercent‚71.59600836534314 disk.drbd1777.in‚0# disk.dm-0.out‚17986.41357421875 disk.dm-0.in‚ 710.1015625( disk.nvme0c0n1.out‚17986.41357421875 disk.sda2.out‚0ûã#E2ñ- _wandbruntime‚2214  _runtime‚2214ÖûO1ÆòÀ $ _timestamp‚1.788409604250032e+09 _runtime‚2214.662133986 _step‚242 & lr-AdamW/pg1‚9.443710005084537e-05 & lr-AdamW/pg2‚9.443710005084537e-05 trainer/global_step‚2999{Ÿ7ËËóÅ $ _timestamp‚1.788409604250032e+09 _runtime‚2214.662133986 _step‚242 & lr-AdamW/pg1‚9.443710005084537e-05 & lr-AdamW/pg2‚9.443710005084537e-05 trainer/global_step‚2999ògzˆš2ô- _wandbruntime‚2214  _runtime‚2214D}±ÓÆ'õÀO _step‚243 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39335479 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4297613619712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39335477 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93517488164864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487936 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89220083033088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60511198 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232428032 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚84306592073169 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚80095002694656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39335477 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚84197752821760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚88421509516800 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚89220106973184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚526 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚2999 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚80095019734016 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4211572339153 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚84197752821760 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _runtime‚2214.663313098 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99846677 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17025821 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60491161 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚84306409912409 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚88421509516800 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4223738692096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4297405131776 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 $ _timestamp‚1.788409604250797e+09 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88421322514432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400ãcågË'öÅO SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88421322514432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 _step‚243 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39335479 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4297613619712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39335477 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93517488164864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487936 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89220083033088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60511198 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232428032 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚84306592073169 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚80095002694656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39335477 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚84197752821760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚88421509516800 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚89220106973184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚526 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚2999 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚80095019734016 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4211572339153 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚84197752821760 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _runtime‚2214.663313098 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99846677 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17025821 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60491161 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚84306409912409 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚88421509516800 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4223738692096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4297405131776 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 $ _timestamp‚1.788409604250797e+09 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039óÃqÂ2÷- _wandbruntime‚2215  _runtime‚2215¡®„éq&øËM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99859084 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4212961503905 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60510948 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚88449985017344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39348138 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚80120962916352 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚88450172033024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4225131843584 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99879149 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚88450172033024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17031211 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚80120945876992 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39348138 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39996280 trainer/global_step‚2999 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4224962830848 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60510948 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60531011 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242900480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4299035079680 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93548055001088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84333924420257 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 % _timestamp‚1.7884096049255252e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84225022186496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39996794 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39348136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚89249019921408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22965581 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99879149 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4212796382297 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _runtime‚2215.338404699 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99859084 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39348136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84333742259289 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _runtime‚2215.338404699 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99859084 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39348136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84333742259289 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60510948 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚88449985017344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39348138 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚80120962916352 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚88450172033024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4225131843584 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99879149 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚88450172033024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17031211 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚80120945876992 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39348138 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39996280 trainer/global_step‚2999 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4224962830848 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60510948 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60531011 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242900480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4299035079680 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93548055001088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84333924420257 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 % _timestamp‚1.7884096049255252e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84225022186496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39996794 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39348136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚89249019921408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22965581 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99879149 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4212796382297ôAò=¼2ú- _wandbruntime‚2215  _runtime‚2215dþÕ±TûÎ 1loss/SRqqt3/assignment_loss‚2.073996247516738 /loss/FBt3/detection_loss‚1.3430709177157825 _step‚245 epoch‚2 % _timestamp‚1.7884096049279227e+09 0loss/FBt2/assignment_loss‚0.5396487950929441 .loss/FRt2/detection_loss‚2.443318834569719 .loss/FRt4/detection_loss‚2.193133898850145 _runtime‚2215.339985378 /loss/FRt3/detection_loss‚2.1472998406642523 1loss/SRqqt2/detection_loss‚1.5812749832104414 2loss/SRqqt1/assignment_loss‚2.2395505796779287 2loss/SRqqt4/assignment_loss‚1.9179411332892335 /loss/FBt2/detection_loss‚3.5221031109491983 .loss/FRt1/detection_loss‚2.790906165876696 /loss/FRt1/assignment_loss‚4.630491825842088 0loss/FRt2/assignment_loss‚3.9090649869706895 0loss/FRt3/assignment_loss‚2.6069883031722827 /loss/FRt4/assignment_loss‚4.153950375729594 .loss/FBt1/detection_loss‚3.721418756705064 .loss/FBt4/detection_loss‚3.344533594755026 1loss/SRqqt2/assignment_loss‚1.715777021761124 1loss/FBt3/assignment_loss‚0.27736622869043154 /loss/FBt4/assignment_loss‚0.468316146705547 0loss/SRqqt1/detection_loss‚2.833387373974829 1loss/SRqqt4/detection_loss‚2.0729308977075247 0loss/FBt1/assignment_loss‚0.5028707257592592 1loss/SRqqt3/detection_loss‚1.7778538162647575 $loss/total_loss‚54.8071945614513 trainer/global_step‚2999“áÝäYüÓ epoch‚2 % _timestamp‚1.7884096049279227e+09 0loss/FBt2/assignment_loss‚0.5396487950929441 .loss/FRt2/detection_loss‚2.443318834569719 .loss/FRt4/detection_loss‚2.193133898850145 _runtime‚2215.339985378 /loss/FRt3/detection_loss‚2.1472998406642523 1loss/SRqqt2/detection_loss‚1.5812749832104414 2loss/SRqqt1/assignment_loss‚2.2395505796779287 2loss/SRqqt4/assignment_loss‚1.9179411332892335 /loss/FBt2/detection_loss‚3.5221031109491983 .loss/FRt1/detection_loss‚2.790906165876696 /loss/FRt1/assignment_loss‚4.630491825842088 0loss/FRt2/assignment_loss‚3.9090649869706895 0loss/FRt3/assignment_loss‚2.6069883031722827 /loss/FRt4/assignment_loss‚4.153950375729594 .loss/FBt1/detection_loss‚3.721418756705064 .loss/FBt4/detection_loss‚3.344533594755026 1loss/SRqqt2/assignment_loss‚1.715777021761124 1loss/FBt3/assignment_loss‚0.27736622869043154 /loss/FBt4/assignment_loss‚0.468316146705547 0loss/SRqqt1/detection_loss‚2.833387373974829 1loss/SRqqt4/detection_loss‚2.0729308977075247 0loss/FBt1/assignment_loss‚0.5028707257592592 1loss/SRqqt3/detection_loss‚1.7778538162647575 $loss/total_loss‚54.8071945614513 trainer/global_step‚2999 1loss/SRqqt3/assignment_loss‚2.073996247516738 /loss/FBt3/detection_loss‚1.3430709177157825 _step‚245õ0Èeþ‘ý‚(:† ‰îãÔ‘ö˜™ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.584' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5168 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™ˆÜ þ‚(:Œ ‰îãÔ²Çèœ) disk./.usagePercent‚71.66132638363675 disk.sda1.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚18181.87939453125$ disk./.usageGB‚1067.805549621582& disk.nvme0n1.out‚18181.87939453125# disk.dm-0.out‚18181.87939453125 disk.rbd3.in‚0 disk.dm-1.out‚97.96875 disk.rbd2.out‚ 51.4404296875 disk.rbd3.out‚0 disk.sda2.in‚0$ memory_percent‚99.21774864196777 disk.sda1.out‚0' proc.memory.availableMB‚ 128.1640625" proc.memory.rssMB‚ 2895.2578125 disk.drbd1777.in‚0 disk.dm-0.in‚ 710.1015625 disk.rbd0.out‚0 cpu‚25.338283746979364 network.sent‚ 6.073621e+06 network.recv‚844747 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.sda3.out‚97.96875 disk.sda.in‚0 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.2890625 disk.nvme0n1.in‚ 710.1015625 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 710.1015625 disk.sda.out‚97.96875) proc.memory.percent‚17.67125129699707”q¢›‘ÿ‚(:† ˜îãÔò憙 gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.553' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1106 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_xö“€‚(:Œ ˜îãÔÜ‹¥œ disk.nvme0n1.in‚ 710.1015625 disk.rbd0.in‚0 disk.rbd2.in‚0( disk.nvme0c0n1.out‚20178.32080078125$ memory_percent‚99.21789169311523 disk.drbd1777.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0& proc.memory.availableMB‚ 128.140625 proc.cpu.threads‚17 network.recv‚850768 disk.sda2.out‚0 disk.sda.in‚0) disk./.usagePercent‚71.76389964234315# disk.dm-0.out‚20178.32080078125 disk.dm-1.in‚0 disk.rbd2.out‚ 52.9521484375) proc.memory.percent‚17.67125129699707 cpu‚25.336382001671023 disk.rbd1.out‚ 1.2890625 disk.dm-1.out‚98.34375 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 710.1015625 disk.sda2.in‚0 disk.sda3.out‚98.34375 disk.sda.out‚98.34375" proc.memory.rssMB‚ 2895.2578125 disk.sda1.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 710.1015625 network.sent‚ 6.181715e+06% disk./.usageGB‚1069.3339653015137 disk.drbd1777.out‚0& disk.nvme0n1.out‚20178.32080078125 disk.rbd3.in‚0 disk.sda1.out‚0[®ô‘‚(:† §îãÔÒï±™ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚105.688' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.1376 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨s°o"‚‚(:— §îãÔËôÐœ cpu‚25.334485512016364 disk.rbd3.out‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 710.1015625 proc.cpu.threads‚17) disk./.usagePercent‚71.88957015679622! disk.nvme0c0n1.in‚ 710.1015625 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0$ memory_percent‚99.21495914459229 network.sent‚ 6.186272e+06 network.recv‚853459 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 710.1015625& disk.nvme0n1.out‚22257.72705078125" proc.memory.rssMB‚ 2895.2578125 disk.rbd3.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 99.01953125 disk.rbd0.in‚0 disk.rbd2.out‚ 53.0458984375 disk.sda.in‚0 disk.sda.out‚ 99.01953125 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.2890625 disk.rbd0.out‚0) proc.memory.percent‚17.67125129699707% disk./.usageGB‚1071.2065467834473 disk.sda3.out‚ 99.01953125 disk.rbd2.in‚0( disk.nvme0c0n1.out‚22257.72705078125# disk.dm-0.out‚22257.72705078125( proc.memory.availableMB‚ 128.62109375öyø„2ƒ- _wandbruntime‚2250  _runtime‚2250lˆœÃÅ„¿ & lr-AdamW/pg1‚9.407219170592006e-05 & lr-AdamW/pg2‚9.407219170592006e-05 trainer/global_step‚3049 # _timestamp‚1.78840963970574e+09 _runtime‚2250.117854329 _step‚246ˆýœ×Ê…Ä # _timestamp‚1.78840963970574e+09 _runtime‚2250.117854329 _step‚246 & lr-AdamW/pg1‚9.407219170592006e-05 & lr-AdamW/pg2‚9.407219170592006e-05 trainer/global_step‚3049öjàS[2†- _wandbruntime‚2250  _runtime‚2250͇ßË'‡ÅO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39968429 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚85561221058560 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4293410014720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚89854462073856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23327056 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101472329 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4293241015296 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61483863 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17296729 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238719488 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39968427 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4293410014720 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚85673040847617 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚85561239061504 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚85561239061504 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4281044157561 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17297278 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61503900 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884096397064939e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚89854462073856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61503900 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚85673223008377 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4280879036161 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚2250.118804759 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101452290 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95034021302784 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _step‚247 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4293241015296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚89854649076224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40624334 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23327054 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214779392 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚81392178850816 trainer/global_step‚3049”oke³ˆÊO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884096397064939e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚89854462073856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61503900 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚85673223008377 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4280879036161 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚2250.118804759 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101452290 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95034021302784 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _step‚247 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4293241015296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚89854649076224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40624334 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23327054 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214779392 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚81392178850816 trainer/global_step‚3049 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39968429 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚85561221058560 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4293410014720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚89854462073856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23327056 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 RDDeviceStatsMonitor.on_train_batch_start/acti‡öwove_bytes.small_pool.peak‚ 410711552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101472329 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4293241015296 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61483863 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17296729 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238719488 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39968427 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4293410014720 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚85673040847617 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚85561239061504 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚85561239061504 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4281044157561 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17297278 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61503900 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217÷LÞà2‰- _wandbruntime‚2250  _runtime‚2250ñdîöÔ&ŠÎM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61523713 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4294634153472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40637199 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884096403622456e+09 _runtime‚ 2250.77487633 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚95064357807104 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚81418122033152 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23334433 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101484736 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39981088 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚85700555355465 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚85588508426240 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚85588490423296 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _step‚248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101504801 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90694643392512 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101504801 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39981086 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚89883124576768 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3049 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17302254 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4282433322313 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚85588508426240 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90694667332608 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39981088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61503650 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚89883311592448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4294634153472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40636685 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚85588490423296 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39981086 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4282268200705 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23334431 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4294803166208 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199cJBKÙ&‹ÓM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4294803166208 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61523713 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4294634153472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40637199 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884096403622456e+09 _runtime‚ 2250.77487633 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚95064357807104 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚81418122033152 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23334433 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101484736 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39981088 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚85700555355465 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚85588508426240 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚85588490423296 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _step‚248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101504801 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90694643392512 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101504801 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39981086 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚89883124576768 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3049 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17302254 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4282433322313 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚85588508426240 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90694667332608 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39981088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61503650 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚89883311592448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4294634153472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40636685 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚85588490423296 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39981086 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4282268200705 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23334431 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416øÕu?2Œ- _wandbruntime‚2250  _runtime‚2250å)<åZÔ 1loss/FBt3/assignment_loss‚0.32632121370045825 .loss/FRt3/detection_loss‚2.254571334567181 % _timestamp‚1.7884096403640587e+09 1loss/SRqqt4/detection_loss‚1.8712299092018858 /loss/FBt2/detection_loss‚2.9160836338996887 %loss/total_loss‚53.01740220567653 trainer/global_step‚3049 .loss/FRt1/detection_loss‚2.694224246037312 1loss/SRqqt2/detection_loss‚1.7422234378230403 /loss/FRt4/assignment_loss‚4.342005607409355 2loss/SRqqt3/assignment_loss‚1.8028842316202398 1loss/FBt4/assignment_loss‚0.40430540834342155 1loss/SRqqt1/assignment_loss‚2.499830289320512 0loss/FBt2/assignment_loss‚0.3834748439490795 1loss/SRqqt1/detection_loss‚2.7216718142682854 1loss/SRqqt3/detection_loss‚1.5444962435735847 .loss/FBt1/detection_loss‚4.936487138271332 /loss/FBt3/detection_loss‚1.0431632105064514 /loss/FRt4/detection_loss‚2.2825604937015433 /loss/FBt4/detection_loss‚2.7279738187789917 epoch‚2 /loss/FRt2/detection_loss‚2.4184597354868185 /loss/FRt1/assignment_loss‚4.464043342150175 0loss/FRt2/assignment_loss‚3.0600683326306553 0loss/FRt3/assignment_loss‚3.1177930346755094 0loss/SRqqt2/assignment_loss‚1.51512716920443 1loss/FBt1/assignment_loss‚0.23424615678636151 2loss/SRqqt4/assignment_loss‚1.7141575597702188 _runtime‚2250.776126646 _step‚249ŽÛÚ¶_ŽÙ 1loss/FBt3/assignment_loss‚0.32632121370045825 .loss/FRt3/detection_loss‚2.254571334567181 % _timestamp‚1.7884096403640587e+09 1loss/SRqqt4/detection_loss‚1.8712299092018858 /loss/FBt2/detection_loss‚2.9160836338996887 %loss/total_loss‚53.01740220567653 trainer/global_step‚3049 .loss/FRt1/detection_loss‚2.694224246037312 1loss/SRqqt2/detection_loss‚1.7422234378230403 /loss/FRt4/assignment_loss‚4.342005607409355 2loss/SRqqt3/assignment_loss‚1.8028842316202398 1loss/FBt4/assignment_loss‚0.40430540834342155 1loss/SRqqt1/assignment_loss‚2.499830289320512 0loss/FBt2/assignment_loss‚0.3834748439490795 1loss/SRqqt1/detection_loss‚2.7216718142682854 1loss/SRqqt3/detection_loss‚1.5444962435735847 .loss/FBt1/detection_loss‚4.936487138271332 /loss/FBt3/detection_loss‚1.0431632105064514 /loss/FRt4/detection_loss‚2.2825604937015433 /loss/FBt4/detection_loss‚2.7279738187789917 epoch‚2 /loss/FRt2/detection_loss‚2.4184597354868185 /loss/FRt1/assignment_loss‚4.464043342150175 0loss/FRt2/assignment_loss‚3.0600683326306553 0loss/FRt3/assignment_loss‚3.1177930346755094 0loss/SRqqt2/assignment_loss‚1.51512716920443 1loss/FBt1/assignment_loss‚0.23424615678636151 2loss/SRqqt4/assignment_loss‚1.7141575597702188 _runtime‚2250.776126646 _step‚249ùÌDƒy‘‚(:† ¶îãÔ÷“™ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.687' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1374 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í+Þ•‚(:’ ¶îãÔ¸ disk.sda3.in‚0 disk.sda.out‚ 99.44921875 disk.sda1.in‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚23547.57763671875 cpu‚25.3326108293537 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd0.out‚0) proc.memory.percent‚17.67125129699707 network.recv‚858159# disk.dm-0.out‚23547.57763671875 disk.dm-1.out‚ 99.44921875 disk.rbd2.out‚ 53.0458984375" proc.memory.rssMB‚ 2895.2578125 network.sent‚ 6.192471e+06) disk./.usagePercent‚72.02607772618123 disk.drbd1777.out‚0 disk.dm-0.in‚ 710.1015625 disk.sda2.in‚0 disk.sda2.out‚0" memory_percent‚99.216628074646( proc.memory.availableMB‚ 128.34765625 disk.nvme0n1.in‚ 710.1015625 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda3.out‚ 99.44921875 disk.sda.in‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚23547.57763671875 disk.dm-1.in‚0$ disk./.usageGB‚1073.240608215332 disk.rbd1.out‚ 1.2890625 disk.rbd3.out‚0! disk.nvme0c0n1.in‚ 710.1015625†ãÅœ‘‚(:‘ ÅîãԨĩ™ gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚108.115' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.622999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë`²¼ ’‚(:• ÅîãÔ½µ±œ network.recv‚864964) disk./.usagePercent‚72.16925608388593 disk.sda1.out‚0& disk.nvme0n1.out‚25695.93310546875" proc.memory.rssMB‚ 2895.2578125 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0# disk.dm-0.out‚25695.93310546875 disk.rbd3.in‚0% disk./.usageGB‚1075.3740692138672 disk.sda2.out‚0 disk.sda.in‚0 disk.drbd1777.out‚0 disk.dm-1.out‚ 99.9921875! disk.nvme0c0n1.in‚ 710.1015625 disk.sda3.out‚ 99.9921875 disk.rbd2.out‚53.27294921875 disk.sda3.in‚0 disk.sda.out‚ 99.9921875 disk.rbd1.out‚ 1.2890625 disk.nvme0n1.in‚ 710.1015625 disk.rbd0.in‚0 disk.rbd3.out‚0$ memory_percent‚99.21877384185791 network.sent‚ 6.301092e+06 disk.sda1.in‚0 disk.dm-0.in‚ 710.1015625( proc.memory.availableMB‚ 127.99609375) proc.memory.percent‚17.67125129699707 cpu‚25.330787952257236 proc.cpu.threads‚17( disk.nvme0c0n1.out‚25695.93310546875 disk.dm-1.in‚0 disk.rbd2.in‚0qØZ2“- _wandbruntime‚2285  _runtime‚2285™øØÆ”À _step‚250 & lr-AdamW/pg1‚9.369820660509154e-05 & lr-AdamW/pg2‚9.369820660509154e-05 trainer/global_step‚3099 $ _timestamp‚1.788409675164167e+09 _runtime‚2285.576306621ÄmË•Å & lr-AdamW/pg1‚9.369820660509154e-05 & lr-AdamW/pg2‚9.369820660509154e-05 trainer/global_step‚3099 $ _timestamp‚1.788409675164167e+09 _runtime‚2285.576306621 _step‚250ú™ès…2–- _wandbruntime‚2285  _runtime‚2285õ.ÔÒJ—ÅO JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚3099 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚86924689295360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62476565 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82689337967616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚86924707298304 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚91287788635648 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚87039671782825 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4362912337920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4439429929984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚2285.577273273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚92111377855488 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62476565 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚86924689295360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40601377 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41262721 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23695906 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103077942 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40601379 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41262193 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40601379 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17566289 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200099328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚86924707298304 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23695904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96550607686144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚91287601633280 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4363081337344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17566815 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚526 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4439229830656 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23695906 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103077942 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40601379 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41262193 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40601379 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17566289 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200099328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚86924707298304 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23695904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96550607686144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚91287601633280 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4363081337344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17566815 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚526 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4439229830656 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚3099 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚86924689295360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62476565 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82689337967616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚86924707298304 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚91287788635648 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚87039671782825 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0û X^2™- _wandbruntime‚2286  _runtime‚2286ÄrøXÔ&šÎM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40614036 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17571568 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚86951958660096 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103130453 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40614038 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17572084 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4440646296576 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚91316451151872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91316451151872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414720 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚82715281149952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62496352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103130453 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚516 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40614038 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 $ _timestamp‚1.788409675836703e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4351905140721 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _runtime‚2286.249533395 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40614036 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚91316264136192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4364474488832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚92140290803712 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚87067004129705 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62516415 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4364305476096 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103110388 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚86951976663040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92140314743808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86951976663040 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _step‚252 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41274849Ÿ®† ›ÓM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86951976663040 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _step‚252 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41274849 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviV§­òSceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40614036 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17571568 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚86951958660096 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103130453 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40614038 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17572084 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4440646296576 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚91316451151872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91316451151872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414720 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚82715281149952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62496352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103130453 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚516 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40614038 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 $ _timestamp‚1.788409675836703e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4351905140721 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _runtime‚2286.249533395 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40614036 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚91316264136192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4364474488832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚92140290803712 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚87067004129705 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62516415 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4364305476096 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103110388 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚86951976663040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92140314743808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0üÁõ×2œ- _wandbruntime‚2286  _runtime‚22862; ;VÐ $ _timestamp‚1.788409675838913e+09 /loss/FRt2/assignment_loss‚4.143819516559817 1loss/SRqqt1/assignment_loss‚2.667075479403138 1loss/SRqqt3/assignment_loss‚2.389389860576817 1loss/FBt3/assignment_loss‚0.30800073322490684 trainer/global_step‚3099 0loss/SRqqt2/detection_loss‚1.380167069897223 /loss/FBt2/detection_loss‚2.8656331151723915 1loss/SRqqt4/assignment_loss‚2.086647256380982 0loss/FBt2/assignment_loss‚0.2172592054944289 /loss/FRt1/detection_loss‚2.9205438967408806 .loss/FBt1/detection_loss‚3.203759661742619 0loss/FBt1/assignment_loss‚0.7127333178201037 .loss/FRt3/detection_loss‚2.159383307231797 1loss/SRqqt4/detection_loss‚1.7641443692975574 0loss/SRqqt1/detection_loss‚2.657150572165847 /loss/FBt3/detection_loss‚1.0439333467958924 /loss/FBt4/detection_loss‚2.6009047282369515 0loss/FRt3/assignment_loss‚2.3599925796190897 2loss/SRqqt2/assignment_loss‚1.9395948891643058 1loss/FBt4/assignment_loss‚0.48588357251566394 .loss/FRt2/detection_loss‚2.183994419732184 1loss/SRqqt3/detection_loss‚1.7409701569094544 epoch‚2 _step‚253 /loss/FRt4/detection_loss‚2.1460326969623567 /loss/FRt1/assignment_loss‚5.814563447031482 0loss/FRt4/assignment_loss‚3.9455027556419373 %loss/total_loss‚53.73707995431783 _runtime‚2286.251066964·®ùø[žÕ 0loss/SRqqt1/detection_loss‚2.657150572165847 /loss/FBt3/detection_loss‚1.0439333467958924 /loss/FBt4/detection_loss‚2.6009047282369515 0loss/FRt3/assignment_loss‚2.3599925796190897 2loss/SRqqt2/assignment_loss‚1.9395948891643058 1loss/FBt4/assignment_loss‚0.48588357251566394 .loss/FRt2/detection_loss‚2.183994419732184 1loss/SRqqt3/detection_loss‚1.7409701569094544 epoch‚2 _step‚253 /loss/FRt4/detection_loss‚2.1460326969623567 /loss/FRt1/assignment_loss‚5.814563447031482 0loss/FRt4/assignment_loss‚3.9455027556419373 %loss/total_loss‚53.73707995431783 _runtime‚2286.251066964 $ _timestamp‚1.788409675838913e+09 /loss/FRt2/assignment_loss‚4.143819516559817 1loss/SRqqt1/assignment_loss‚2.667075479403138 1loss/SRqqt3/assignment_loss‚2.389389860576817 1loss/FBt3/assignment_loss‚0.30800073322490684 trainer/global_step‚3099 0loss/SRqqt2/detection_loss‚1.380167069897223 /loss/FBt2/detection_loss‚2.8656331151723915 1loss/SRqqt4/assignment_loss‚2.086647256380982 0loss/FBt2/assignment_loss‚0.2172592054944289 /loss/FRt1/detection_loss‚2.9205438967408806 .loss/FBt1/detection_loss‚3.203759661742619 0loss/FBt1/assignment_loss‚0.7127333178201037 .loss/FRt3/detection_loss‚2.159383307231797 1loss/SRqqt4/detection_loss‚1.7641443692975574ý›‡j\‘Ÿ‚(:† ÔîãÔˆ£¯™ gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.917' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3834 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒS¯ ‚(: ÔîãÔ—¤á› disk.sda1.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.rbd2.out‚53.27294921875 disk.rbd3.out‚0$ memory_percent‚99.21913146972656 network.sent‚ 6.306499e+06 disk.sda.out‚ 100.671875 disk.rbd1.out‚ 1.2890625 disk.rbd0.in‚0" proc.memory.rssMB‚ 2895.2578125) proc.memory.percent‚17.67125129699707) disk./.usagePercent‚72.28242523842698 disk.drbd1777.in‚0& disk.nvme0n1.out‚28326.98779296875 disk.dm-0.in‚ 710.1015625% disk./.usageGB‚1077.0603713989258 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 710.1015625# disk.dm-0.out‚28326.98779296875 disk.rbd2.in‚0 disk.sda3.out‚ 100.671875( disk.nvme0c0n1.out‚28326.98779296875$ proc.memory.availableMB‚127.9375 cpu‚25.32909395508127 network.recv‚868959 disk.dm-1.in‚0 disk.dm-1.out‚ 100.671875! disk.nvme0c0n1.in‚ 710.1015625 disk.sda2.in‚0ì¸ö‘¡‚(:† ãîãÔ¨ú»š gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.543' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1086 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ‰ó¢‚(:Ž ãîãԌҚ disk.rbd3.out‚0( disk.nvme0c0n1.out‚29792.13623046875 disk.sda3.in‚0& disk.nvme0n1.out‚29792.13623046875 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚101.4375 disk.sda.in‚0$ memory_percent‚99.21953678131104" proc.memory.rssMB‚ 2895.2578125! disk.nvme0c0n1.in‚ 710.1015625 disk.sda3.out‚101.4375 disk.nvme0n1.in‚ 710.1015625$ disk./.usageGB‚1079.061321258545) disk./.usagePercent‚72.41671066241479 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚54.98779296875 disk.rbd3.in‚0 disk.sda2.in‚0) proc.memory.percent‚17.67125129699707 disk.dm-0.in‚ 710.1015625 disk.sda.out‚101.4375 cpu‚25.327268570640413 disk.sda2.out‚0 disk.drbd1777.in‚0 network.recv‚875245 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 1.2890625( proc.memory.availableMB‚ 127.87109375 proc.cpu.threads‚17 disk.sda1.in‚0 network.sent‚ 6.414706e+06# disk.dm-0.out‚29792.13623046875 disk.rbd0.out‚0%Ùº^2£- _wandbruntime‚2320  _runtime‚2320gòBÅǤÁ % _timestamp‚1.7884097101444805e+09 _runtime‚2320.556610835 _step‚254 & lr-AdamW/pg1‚9.331522268629426e-05 & lr-AdamW/pg2‚9.331522268629426e-05 trainer/global_step‚3149åh]Ì¥Æ % _timestamp‚1.7884097101444805e+09 _runtime‚2320.556610835 _step‚254 & lr-AdamW/pg1‚9.331522268629426e-05 & lr-AdamW/pg2‚9.331522268629426e-05 trainer/global_step‚3149þ¹@2¦- _wandbruntime‚2320  _runtime‚23206B0œÊ'§ÄO WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63489304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚93557049206784 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4510325553152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚88288175535104 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98067374759936 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚83986497084416 $ _timestamp‚1.788409710145196e+09 _runtime‚2320.557530168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚92720741192704 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4432583660544 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚98067133943296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88406302718033 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4419822672977 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63469267 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24064756 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 trainer/global_step‚3149 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41234327 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚506 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63489304 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4432752659968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816640 _step‚255 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92720741192704 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚88406484878793 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41234329 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24064754 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41234327 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104723633 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚92720928195072 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚88288157532160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41902485 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104703594 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚92720928195072 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17837729 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93557025266688 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4510108676608 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚83986480045056 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4432583660544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4419987794377 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41901977 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88406302718033 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4419822672977 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63469267 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24064756 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 trainer/global_step‚3149 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41234327 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚506 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63489304 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4432752659968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816640 _step‚255 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92720741192704 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚88406484878793 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41234329 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24064754 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41234327 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104723633 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚92720928195072 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚88288157532160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41902485 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104703594 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚92720928195072 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17837729 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93557025266688 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4510108676608 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚83986480045056 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4432583660544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4419987794377 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41901977 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚515 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17842506 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4421211837521 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚256 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93585938214912 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84012423227392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41246986 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚98097699847680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚88315426896896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3149 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41914637 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41246986 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98097463238656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚92749590711296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93585962155008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚92749403695616 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104756105 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚515 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17842506 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4421211837521 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚256 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93585938214912 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84012423227392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41246986 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚98097699847680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚88315426896896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3149 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41914637 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41246986 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98097463238656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚92749590711296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93585962155008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚92749403695616 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104756105 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚89651643771904 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42543741 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95002672677888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4489459612785 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4502254983168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41867277 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24433606 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106329246 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚2355.628194886 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64482006 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 [IDeviceStatsMonitor.on_train_1]Ybatch_start/active_bytes.small_pool.allocated‚ 4502423982592 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4489294491385 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚94154067754496 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24433604 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚94153880752128 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚89773115814001 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4581260145152 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚95002696617984 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚94153880752128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚89651625768960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106349285 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 trainer/global_step‚3199 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚89651643771904 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4502254983168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚89651643771904 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42543741 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95002672677888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4489459612785 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4502254983168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41867277 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24433606 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106329246 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚2355.628194886 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64482006 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4502423982592 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4489294491385 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚94154067754496 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24433604 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚94153880752128 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚89773115814001 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4581260145152 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚95002696617984 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚94153880752128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚89651625768960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106349285 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 trainer/global_step‚3199ƒ9@,2»- _wandbruntime‚2356  _runtime‚2356ÿÊÔ&¼ÎM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4503817134080 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚99614050969600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95031609566208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚89800448161089 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41879936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18115767 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18115243 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64501819 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚95031585626112 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206377472 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚89678895133696 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106381757 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42556224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4582671720960 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24440983 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚94182730270720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4503648121344 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚99614281287168 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚94182730270720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42556750 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24440981 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚85309582344192 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106381757 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41879938 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41879938 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4582465343488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚85309599383552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64501819 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64481756 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4490683655929 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 trainer/global_step‚3199 _step‚260 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4503648121344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41879936 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106381757 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42556224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4582671720960 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24440983 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚94182730270720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4503648121344 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚99614281287168 VCDeviceStatsMonitor.on_train_batch_enhµ¦Žb d/allocated_bytes.all.allocated‚94182730270720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42556750 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24440981 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚85309582344192 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106381757 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41879938 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41879938 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4582465343488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚85309599383552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64501819 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64481756 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4490683655929 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 trainer/global_step‚3199 _step‚260 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4503648121344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552„¥Ù¯ó2¾- _wandbruntime‚2356  _runtime‚2356Âh>S¿Í .loss/FRt4/detection_loss‚2.483392221480608 1loss/SRqqt4/detection_loss‚2.0981789471172707 /loss/FBt3/detection_loss‚0.8326264997746156 /loss/FBt3/assignment_loss‚0.314861185934357 0loss/FBt4/assignment_loss‚0.6630504533646412 0loss/FBt1/assignment_loss‚0.3332586266077279 1loss/SRqqt1/detection_loss‚2.7831307848294577 /loss/SRqqt3/detection_loss‚1.75744807596008 _step‚261 /loss/FRt2/assignment_loss‚3.605414286492363 2loss/SRqqt3/assignment_loss‚2.2158124574593137 0loss/FBt2/assignment_loss‚0.4918645927085663 /loss/FBt4/detection_loss‚2.4234223677998497 _runtime‚ 2356.27886808 /loss/FRt2/detection_loss‚2.2287967914626714 .loss/FBt2/detection_loss‚2.388563483953479 $loss/total_loss‚53.0688022206494 epoch‚2 /loss/FRt1/assignment_loss‚4.382193005603293 2loss/SRqqt2/assignment_loss‚1.5720400624247133 .loss/FRt1/detection_loss‚3.322914312715116 /loss/FRt3/detection_loss‚2.4414851826926074 /loss/FBt1/detection_loss‚2.9803123019990467 2loss/SRqqt1/assignment_loss‚2.3163006216287614 1loss/SRqqt2/detection_loss‚1.6519581362389135 trainer/global_step‚3199 $ _timestamp‚1.788409745866819e+09 0loss/FRt3/assignment_loss‚3.8757756228248277 /loss/FRt4/assignment_loss‚4.037781465053558 2loss/SRqqt4/assignment_loss‚1.8682207345235637®t ½XÀÒ /loss/FRt1/assignment_loss‚4.382193005603293 2loss/SRqqt2/assignment_loss‚1.5720400624247133 .loss/FRt1/detection_loss‚3.322914312715116 /loss/FRt3/detection_loss‚2.4414851826926074 /loss/FBt1/detection_loss‚2.9803123019990467 2loss/SRqqt1/assignment_loss‚2.3163006216287614 1loss/SRqqt2/detection_loss‚1.6519581362389135 trainer/global_step‚3199 $ _timestamp‚1.788409745866819e+09 0loss/FRt3/assignment_loss‚3.8757756228248277 /loss/FRt4/assignment_loss‚4.037781465053558 2loss/SRqqt4/assignment_loss‚1.8682207345235637 .loss/FRt4/detection_loss‚2.483392221480608 1loss/SRqqt4/detection_loss‚2.0981789471172707 /loss/FBt3/detection_loss‚0.8326264997746156 /loss/FBt3/assignment_loss‚0.314861185934357 0loss/FBt4/assignment_loss‚0.6630504533646412 0loss/FBt1/assignment_loss‚0.3332586266077279 1loss/SRqqt1/detection_loss‚2.7831307848294577 /loss/SRqqt3/detection_loss‚1.75744807596008 _step‚261 /loss/FRt2/assignment_loss‚3.605414286492363 2loss/SRqqt3/assignment_loss‚2.2158124574593137 0loss/FBt2/assignment_loss‚0.4918645927085663 /loss/FBt4/detection_loss‚2.4234223677998497 _runtime‚ 2356.27886808 /loss/FRt2/detection_loss‚2.2287967914626714 .loss/FBt2/detection_loss‚2.388563483953479 $loss/total_loss‚53.0688022206494 epoch‚2…'mª‘Á‚(:† ŸïãÔðàÓ™ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.324' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4648 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦,Qò‚(:ˆ ŸïãÔŠ§· disk.sda2.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda3.out‚ 103.84375( proc.memory.availableMB‚ 127.53515625 disk.dm-1.in‚0 disk.rbd2.out‚ 55.16796875 disk.sda3.in‚0 network.recv‚894392 disk.nvme0n1.in‚ 710.1015625 disk.dm-1.out‚ 103.84375 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 103.84375 disk.drbd1777.out‚0$ memory_percent‚99.22158718109131) proc.memory.percent‚17.67125129699707 cpu‚25.320211569039685 network.sent‚ 6.540731e+06& disk.nvme0c0n1.out‚38119.150390625 disk.sda.in‚0 disk.drbd1777.in‚0 disk.rbd1.out‚ 1.2890625% disk./.usageGB‚1083.8569259643555$ disk.nvme0n1.out‚38119.150390625 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 710.1015625 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 710.1015625" proc.memory.rssMB‚ 2895.2578125( disk./.usagePercent‚72.7385477180021! disk.dm-0.out‚38119.150390625 disk.rbd3.out‚0£+ ›Ã‚(: ®ïãÔü¦™ gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚80.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚16.057199999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„Ÿ±÷Ä‚(: ®ïãԌଛ network.recv‚901429) disk./.usagePercent‚72.72057419704568 disk.sda.out‚ 104.265625 disk.sda1.in‚0 disk.drbd1777.in‚0) proc.memory.percent‚17.67125129699707 cpu‚25.318575070993024 disk.sda1.out‚0% disk.nvme0n1.out‚39044.2861328125 disk.sda3.in‚0 disk.rbd2.out‚ 55.16796875# memory_percent‚99.2194652557373' proc.memory.availableMB‚ 127.8828125 disk.sda2.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1083.5891075134277 disk.drbd1777.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 710.1015625" disk.dm-0.out‚39044.2861328125 disk.dm-1.out‚ 104.265625 disk.rbd3.in‚0 network.sent‚ 6.649598e+06 disk.sda3.out‚ 104.265625 disk.sda.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 1.2890625 disk.nvme0n1.in‚ 710.1015625! disk.nvme0c0n1.in‚ 710.1015625' disk.nvme0c0n1.out‚39044.2861328125" proc.memory.rssMB‚ 2895.2578125 disk.dm-1.in‚0 ò“2Å- _wandbruntime‚2390  _runtime‚2390åÀîÆÆÀ & lr-AdamW/pg1‚9.252257950659929e-05 & lr-AdamW/pg2‚9.252257950659929e-05 trainer/global_step‚3249 $ _timestamp‚1.788409780466032e+09 _runtime‚2390.878205499 _step‚262zÑ’ËÇÅ & lr-AdamW/pg1‚9.252257950659929e-05 & lr-AdamW/pg2‚9.252257950659929e-05 trainer/global_step‚3249 $ _timestamp‚1.788409780466032e+09 _runtime‚2390.878205499 _step‚262†T¾í2È- _wandbruntime‚2390  _runtime‚2390`J£SÍ'ÉÇO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚91139746749209 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚91139564588449 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚96448320089088 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4651912995840 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65474708 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚95587020311552 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚95587020311552 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 % _timestamp‚1.7884097804669251e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚91015112008704 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4571926305792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43183317 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18381388 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4652127775232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4558931431193 trainer/global_step‚3249 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42500227 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚95587207313920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚91015094005760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24802456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24802454 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18380863 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚101100471804416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42500229 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107974937 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4558766309793 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚95587207313920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4572095305216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚86580798278656 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4571926305792 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65474708 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43183844 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214779392 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42500227 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42500229 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65454671 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238719488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107954898 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚91015094005760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚86580815318016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚2390.879359553 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚91015112008704 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚96448344029184 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160@L÷9Ò'ÊÌO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107954898 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚91015094005760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚86580815318016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚2390.879359553 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚91015112008704 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚96448344029184 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚91139746749209 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚91139564588449 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚96448320089088 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4651912995840 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65474708 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚95587020311552 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚95587020311552 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 % _timestamp‚1.7884097804669251e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚91015112008704 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4571926305792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43183317 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18381388 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4652127775232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4558931431193 trainer/global_step‚3249 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42500227 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚95587207313920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚91015094005760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24802456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24802454 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18380863 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚101100471804416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42500229 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107974937 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4558766309793 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚95587207313920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4572095305216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚86580798278656 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4571926305792 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65474708 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43183844 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214779392 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42500227 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42500229 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65454671 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238719488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0‡9(Gµ2Ë- _wandbruntime‚2391  _runtime‚2391`„˜4 ÌÑM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43196422 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚470 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108007409 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚91042381373440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _step‚264 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42512886 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚91042363370496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 C9DeviceStat#D—á£sMonitor.on_train_batch_end/allocation.all.peak‚24400 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚96477233037312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18386589 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚95615682814464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4573319443968 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414720 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚91167079096297 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 % _timestamp‚1.7884097811347573e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚91042381373440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚472 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43195950 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65494521 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42512888 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚95615869830144 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 _runtime‚2391.547451467 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4653541088256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚3249 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4653332613632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4560320595945 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4560155474337 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24809833 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65494521 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4573488456704 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚91166896935329 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65474458 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4573488456704 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108007409f Ü&ÍÖM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42512886 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43195950 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65494521 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42512888 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚95615869830144 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 _runtime‚2391.547451467 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4653541088256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚3249 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4653332613632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4560320595945 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4560155474337 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24809833 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65494521 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4573488456704 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚91166896935329 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65474458 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4573488456704 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108007409 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43196422 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚470 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108007409 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚91042381373440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _step‚264 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42512886 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚91042363370496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚96477233037312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18386589 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚95615682814464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4573319443968 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414720 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚91167079096297 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 % _timestamp‚1.7884097811347573e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚91042381373440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚472 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 Fƒ»2Õ- _wandbruntime‚2426  _runtime‚2426rË0°ÆÖÀ _runtime‚2426.089503903 _step‚266 & lr-AdamW/pg1‚9.211308543135218e-05 & lr-AdamW/pg2‚9.211308543135218e-05 trainer/global_step‚3299 $ _timestamp‚1.788409815677346e+09²Ë×Å trainer/global_step‚3299 $ _timestamp‚1.788409815677346e+09 _runtime‚2426.089503903 _step‚266 & lr-AdamW/pg1‚9.211308543135218e-05 & lr-AdamW/pg2‚9.211308543135218e-05Š›ÊÏÅ2Ø- _wandbruntime‚2426  _runtime‚2426Á]·û%ÙÆO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43133177 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25171304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43133179 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚87877974434816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224039424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4722769112576 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚92378562242560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43821672 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚97020159870976 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102616736612864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25171306 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4641766627840 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 $ _timestamp‚1.788409815678088e+09 _runtime‚2426.090475233 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66447373 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚92378562242560 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚97020159870976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18650902 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4628403249601 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66467410 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4722969211904 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4628238128201 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102616960652288 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109580550 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚97020346873344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4641597628416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚97020346873344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚92506377684417 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43822208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚267 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43133179 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43133177 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4641597628416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18650368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pËC"âÑool.allocated‚199 trainer/global_step‚3299 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66447373 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92378580245504 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0’'xüÑ'ÚËO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚3299 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66447373 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92378580245504 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43133177 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25171304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43133179 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚87877974434816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224039424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4722769112576 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚92378562242560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43821672 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚97020159870976 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102616736612864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25171306 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4641766627840 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 $ _timestamp‚1.788409815678088e+09 _runtime‚2426.090475233 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66447373 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚92378562242560 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚97020159870976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18650902 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4628403249601 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66467410 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4722969211904 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4628238128201 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102616960652288 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109580550 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚97020346873344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4641597628416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚97020346873344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚92506377684417 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43822208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚267 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43133179 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43133177 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4641597628416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18650368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0‹ï½V2Û- _wandbruntime‚2426  _runtime‚24267è6%×&ÜÑM S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚97049009389568 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109612996 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4643159779328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25178681 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚3299 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43835142 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43145838 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚102647298251264 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4643159779328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4724185388032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚2426.735364304 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97922904388608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4629627292745 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43145838 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚92405831607296 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚92405831607296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66487223 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66487223 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43145836 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚97049009389568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43145836 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚505 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4724393862656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66467160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87903917617152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4642990766592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18655956 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚92533527870537 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43834637 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚87903900577792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109633061 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4629792414353 % _timestamp‚1.7884098163224971e+09 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚92405849610240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚97048822373888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109633061 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4642990766592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25178683 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚92405849610240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18656459¡-iîÜ&ÝÖM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚2426.735364304 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97922904388608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4629627292745 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43145838 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚92405831607296 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚92405831607296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66487223 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66487223 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43145836 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚97049009389568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43145836 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚505 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4724393862656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66467160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87903917617152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4642990766592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18655956 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚92533527870537 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43834637 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚87903900577792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109633061 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4629792414353 % _timestamp‚1.7884098163224971e+09 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚92405849610240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚97048822373888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109633061 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4642990766592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25178683 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚92405849610240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18656459 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚97049009389568 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109612996 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4643159779328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25178681 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚3299 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43835142 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43145838 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚102647298251264 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4643159779328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4724185388032Œs$ýP2Þ- _wandbruntime‚2426  _runtime‚2426ÖôX:YßÓ _step‚269 /loss/FRt4/assignment_loss‚3.951690399646759 2loss/SRqqt3/assignment_loss‚2.0534545117989182 0loss/FBt1/assignment_loss‚0.4548639650674886 .loss/FRt1/detection_loss‚2.969524873154504 &loss/total_loss‚52.473775724888895 _runtime‚2426.736542494 /loss/FRt2/assignment_loss‚3.502162300215827 2loss/SRqqt2/assignment_loss‚1.6565177623182534 1loss/SRqqt2/detection_loss‚1.5095687540248037 1loss/SRqqt3/detection_loss‚1.4894473613239825 1loss/SRqqt4/assignment_loss‚1.990202089978589 0loss/FBt4/assignment_loss‚0.5331903323344401 /loss/FRt1/assignment_loss‚5.175461062363216 0loss/FRt3/assignment_loss‚2.8363827592448185 1loss/SRqqt1/assignment_loss‚2.966577981199537 0loss/SRqqt1/detection_loss‚2.603736447436469 epoch‚2 trainer/global_step‚3299 % _timestamp‚1.7884098163244805e+09 /loss/FRt2/detection_loss‚2.2739961703618365 /loss/FBt1/detection_loss‚2.7612233133543107 /loss/FBt3/detection_loss‚1.2812374158372268 1loss/FBt2/assignment_loss‚0.33977287114136673 0loss/SRqqt4/detection_loss‚1.923956765068902 1loss/FBt3/assignment_loss‚0.27018922680965235 /loss/FRt3/detection_loss‚2.4178118893974707 /loss/FBt4/detection_loss‚2.7858626445134482 /loss/FRt4/detection_loss‚1.9268340998888016 .loss/FBt2/detection_loss‚2.800110728408282\À|¶àØ epoch‚2 trainer/global_step‚3299 % _timestamp‚1.7884098163244805e+09 /loss/FRt2/detection_loss‚2.2739961703618365 /loss/FBt1/detection_loss‚2.7612233133543107 /loss/FBt3/detection_loss‚1.2812374158372268 1loss/FBt2/assignment_loss‚0.33977287114136673 0loss/SRqqt4/detection_loss‚1.923956765068902 1loss/FBt3/assignment_loss‚0.27018922680965235 /loss/FRt3/detection_loss‚2.4178118893974707 /loss/FBt4/detection_loss‚2.7858626445134482 /loss/FRt4/detection_loss‚1.9268340998888016 .loss/FBt2/detection_loss‚2.800110728408282 _step‚269 /loss/FRt4/assignment_loss‚3.951690399646759 2loss/SRqqt3/assignment_loss‚2.053454511798õ›`4¨9182 0loss/FBt1/assignment_loss‚0.4548639650674886 .loss/FRt1/detection_loss‚2.969524873154504 &loss/total_loss‚52.473775724888895 _runtime‚2426.736542494 /loss/FRt2/assignment_loss‚3.502162300215827 2loss/SRqqt2/assignment_loss‚1.6565177623182534 1loss/SRqqt2/detection_loss‚1.5095687540248037 1loss/SRqqt3/detection_loss‚1.4894473613239825 1loss/SRqqt4/assignment_loss‚1.990202089978589 0loss/FBt4/assignment_loss‚0.5331903323344401 /loss/FRt1/assignment_loss‚5.175461062363216 0loss/FRt3/assignment_loss‚2.8363827592448185 1loss/SRqqt1/assignment_loss‚2.966577981199537 0loss/SRqqt1/detection_loss‚2.603736447436469bùrÚ‘á‚(:† ÛïãÔẙ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚112.291' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4582 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uW¾*â‚(:’ ÛïãÔ£éî› disk.sda1.out‚0 disk.dm-1.out‚ 106.02734375 disk.sda.out‚ 106.02734375 disk.rbd0.in‚0$ memory_percent‚99.22220706939697( proc.memory.availableMB‚ 127.43359375) proc.memory.percent‚17.67125129699707 cpu‚25.31353838667588 network.recv‚915597$ disk./.usageGB‚1084.089126586914 disk.rbd3.out‚0 disk.rbd1.out‚ 1.2890625 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 710.1015625 disk.rbd2.in‚0! disk.nvme0c0n1.in‚ 710.1015625 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 710.1015625 disk.rbd2.out‚ 56.8984375) disk./.usagePercent‚72.75413089661129 disk.sda3.in‚0 disk.sda3.out‚ 106.02734375 disk.sda1.in‚0% disk.nvme0n1.out‚39247.7802734375 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.rbd1.in‚0" disk.dm-0.out‚39247.7802734375 disk.rbd0.out‚0 network.sent‚ 6.768943e+06' disk.nvme0c0n1.out‚39247.7802734375 disk.sda.in‚0" proc.memory.rssMB‚ 2895.2578125 disk.sda2.out‚0"µ”œã‚(:‘ êïãÔº ´™ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.356' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.871199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡»jñä‚(: êïãÔО disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚56.98486328125( proc.memory.availableMB‚ 127.69140625% disk.nvme0c0n1.out‚39347.25390625 disk.dm-1.out‚ 106.47265625 disk.rbd3.out‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda3.in‚0 disk.sda.in‚0 proc.cpu.threads‚17 network.sent‚ 6.877677e+06 disk.sda3.out‚ 106.47265625 disk.nvme0n1.in‚ 710.1015625 disk.dm-0.in‚ 710.1015625) proc.memory.percent‚17.67125129699707# disk.nvme0n1.out‚39347.25390625) disk./.usagePercent‚72.75413089661129 disk.sda.out‚ 106.47265625 disk.rbd1.out‚ 1.2890625# memory_percent‚99.2206335067749$ disk./.usageGB‚1084.089126586914 disk.sda2.in‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2895.2578125 cpu‚25.311889398395678 network.recv‚922502 disk.dm-1.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 710.1015625 disk.dm-0.out‚39347.25390625uïü‘å‚(:† ùïãÔ¿¬™ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.903' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7806 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂuùÍ#æ‚(:˜ ùïãÔéåÂœ) disk./.usagePercent‚72.75413601676199 disk.sda.in‚0% disk./.usageGB‚1084.0892028808594 disk.sda.out‚ 107.01953125 disk.nvme0n1.in‚ 710.1015625 disk.rbd3.in‚0" proc.memory.rssMB‚ 2895.2578125! disk.nvme0c0n1.in‚ 710.1015625 disk.sda1.in‚0 disk.drbd1777.out‚0 cpu‚25.310443146935132 proc.cpu.threads‚17( disk.nvme0c0n1.out‚39446.36376953125 disk.drbd1777.in‚0& disk.nvme0n1.out‚39446.36376953125 disk.rbd0.in‚0 disk.rbd2.out‚56.99658203125 network.recv‚926759 disk.sda3.out‚ 107.01953125 disk.rbd1.in‚0 disk.dm-0.in‚ 710.1015625 disk.dm-1.out‚ 107.01953125 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd1.out‚ 1.2890625# disk.dm-0.out‚39446.36376953125 disk.rbd3.out‚0$ memory_percent‚99.22075271606445 network.sent‚ 6.88374e+06 disk.dm-1.in‚0& proc.memory.availableMB‚ 127.671875) proc.memory.percent‚17.671251296997077öF2ç- _wandbruntime‚2461  _runtime‚2461±SÇèÁ & lr-AdamW/pg1‚9.169492287501311e-05 & lr-AdamW/pg2‚9.169492287501311e-05 trainer/global_step‚3349 % _timestamp‚1.7884098509860568e+09 _runtime‚2461.398225488 _step‚270­ìùrÌéÆ _runtime‚2461.398225488 _step‚270 & lr-AdamW/pg1‚9.169492287501311e-05 & lr-AdamW/pg2‚9.169492287501311e-05 trainer/global_step‚3349 % _timestamp‚1.7884098509860568e+09Žµ {2ê- _wandbruntime‚2461  _runtime‚2461Þ°¬Í'ëÇO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44462409 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4711268951040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _runtime‚2461.399275417 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98453299430400 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4697709946609 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4711268951040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44462942 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚93742030479360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4793840679936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚89175133551616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67440075 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25540156 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚99339638851584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚89175116512256 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 trainer/global_step‚3349 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67460112 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67440075 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25540154 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚98453486432768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4793636386304 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚104133479531520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚93873008619625 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚93742048482304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18922786 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚98453486432768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 % _timestamp‚1.7884098509868414e+09 _step‚271 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4697875068009 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚93742048482304 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43766127 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43766129 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚104133251297792 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4711437950464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111226241 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚98453299430400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18922255 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚93872826458865 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204293632 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199åµÒ'ìÌO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67460112 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43766129 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233728 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚98453486432768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4793636386304 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚104133479531520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚93873008619625 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚93742048482304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18922786 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚98453486432768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 % _timestamp‚1.7884098509868414e+09 _step‚271 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4697875068009 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚93742048482304 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43766127 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43766129 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚104133251297792 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4711437950464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111226241 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚98453299430400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18922255 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚93872826458865 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204293632 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44462409 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4711268951040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _runtime‚2461.399275417 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98453299430400 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4697709946609 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4711268951040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44462942 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚93742030479360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4793840679936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚89175133551616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67440075 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25540156 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚99339638851584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚89175116512256 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 trainer/global_step‚3349 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67460112 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67440075 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25540154 BßÀW2í- _wandbruntime‚2462  _runtime‚2462ˆ ÈuîÑM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25547533 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4712662089216 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25547531 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18927669 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚99368551799808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98481961933312 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4795051589632 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18928175 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _step‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111258713 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚104163811864064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚93900158805745 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚506 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚89201059694592 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43778788 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 _runtime‚2462.068232993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43778786 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4712662089216 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 trainer/global_step‚3349 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111238648 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4712831101952 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚93769317847040 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67479925 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚89201076733952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4699264232761 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67459862 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67479925 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4795260064256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67459862 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚93769299844096 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111258713 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚104163579449344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚508 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43778788 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚93769317847040 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98482148948992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 % _timestamp‚1.7884098516555777e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44475200 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚99368527859712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199·‚'YÜ&ïÖM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _step‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111258713 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚104163811864064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚93900158805745 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚506 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚89201059694592 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43778788 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 _runtime‚2462.068232993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43778786 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4712662089216 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414720 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 trainer/global_step‚3349 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111238648 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4712831101952 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚93769317847040 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67479925 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚89201076733952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4699264232761 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67459862 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67479925 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4795260064256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67459862 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚93769299844096 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111258713 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚104163579449344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚508 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43778788 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚93769317847040 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98482148948992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 % _timestamp‚1.7884098516555777e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44475200 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚99368527859712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25547533 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4712662089216 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25547531 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18927669 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚99368551799808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98481961933312 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4795051589632 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18928175Á÷3±2ð- _wandbruntime‚2462  _runtime‚24621¾GXñÒ /loss/FRt1/assignment_loss‚5.426212457900352 -loss/FRt1/detection_loss‚2.51484584935168 /loss/FRt3/detection_loss‚2.5523146969931467 1loss/SRqqt3/detection_loss‚1.6902044395414682 % _timestamp‚1.7884098516574793e+09 2loss/SRqqt3/assignment_loss‚1.7286890624807432 1loss/SRqqt2/detection_loss‚1.4587290474697003 2loss/SRqqt4/assignment_loss‚2.0170390516005714 .loss/FBt2/detection_loss‚2.522661294349215 _runtime‚2462.069534135 2loss/SRqqt1/assignment_loss‚2.4818508327007294 2loss/SRqqt2/assignment_loss‚1.6727975962471358 /loss/FBt3/detection_loss‚0.8478239301415996 trainer/global_step‚3349 /loss/FRt4/assignment_loss‚3.980885304104198 1loss/FBt3/assignment_loss‚0.32977605345224265 0loss/FBt4/assignment_loss‚0.4673397824950977 /loss/FRt2/detection_loss‚2.3682347003449786 1loss/SRqqt1/detection_loss‚2.7011664668718973 _step‚273 /loss/FRt2/assignment_loss‚2.777958672097389 /loss/FRt3/assignment_loss‚2.903348391396659 .loss/FRt4/detection_loss‚2.028719560666518 %loss/total_loss‚51.40956405046924 0loss/FBt2/assignment_loss‚0.5793349561367255 1loss/SRqqt4/detection_loss‚1.9151896736745178 epoch‚2 1loss/FBt1/assignment_loss‚0.43327742284163834 /loss/FBt1/detection_loss‚3.7204156816005707 .loss/FBt4/detection_loss‚2.290749126010471ÊúÆÊ]ò× 1loss/FBt1/assignment_loss‚0.43327742284163834 /loss/FBt1/detection_loss‚3.7204156816005707 .loss/FBt4/detection_loss‚2.290749126010471 /loss/FRt1/assignment_loss‚5.426212457900352 -loss/FRt1/detection_loss‚2.51484584935168 /loss/FRt3/detection_loss‚2.5523146969931467 1loss/SRqqt3/detection_loss‚1.6902044395414682 % _timestamp‚1.7884098516574793e+09 2loss/SRqqt3/assignment_loss‚1.7286890624807432 1loss/SRqqt2/detection_loss‚1.4587290474697003 2loss/SRqqt4/assignment_loss‚2.0170390516005714 .loss/FBt2/detection_loss‚2.522661294349215 _runtime‚2462.069534135 2loss/SRqqt1/assignment_loss‚2.4818508327007294 2loss/SRqqt2/assignment_loss‚1.6727975962471358 /loss/FBt3/detection_loss‚0.8478239301415996 trainer/global_step‚3349 /loss/FRt4/assignment_loss‚3.980885304104198 1loss/FBt3/assignment_loss‚0.32977605345224265 0loss/FBt4/assignment_loss‚0.4673397824950977 /loss/FRt2/detection_loss‚2.3682347003449786 1loss/SRqqt1/detection_loss‚2.7011664668718973 _step‚273 /loss/FRt2/assignment_loss‚2.777958672097389 /loss/FRt3/assignment_loss‚2.903348391396659 .loss/FRt4/detection_loss‚2.028719560666518 %loss/total_loss‚51.40956405046924 0loss/FBt2/assignment_loss‚0.5793349561367255 1loss/SRqqt4/detection_loss‚1.9151896736745178 epoch‚2‘ ˆïŒó‚(: ˆðãÔ‹°©š gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚98.7' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.74 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉðÿÙô‚(: ˆðãÔÊúý› disk.rbd1.out‚1.375! disk.dm-0.out‚39447.232421875& disk.nvme0c0n1.out‚39447.232421875) proc.memory.percent‚17.67125129699707 cpu‚25.308791423416213 disk.sda1.out‚0 disk.drbd1777.out‚0 network.sent‚ 6.889163e+06) disk./.usagePercent‚72.75413345668665 disk.nvme0n1.in‚ 710.1015625 disk.rbd0.in‚0# memory_percent‚99.2225170135498" proc.memory.rssMB‚ 2895.2578125 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 710.1015625 disk.sda3.in‚0 disk.sda3.out‚ 107.46484375% disk./.usageGB‚1084.0891647338867$ disk.nvme0n1.out‚39447.232421875 disk.dm-1.out‚ 107.46484375 disk.rbd2.out‚57.12158203125 disk.sda.in‚0 disk.sda.out‚ 107.46484375 disk.dm-0.in‚ 710.1015625 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda2.out‚0 network.recv‚930563 disk.drbd1777.in‚0 disk.rbd2.in‚0' proc.memory.availableMB‚ 127.3828125 disk.sda2.in‚0 proc.cpu.threads‚17¢…@‘õ‚(:† —ðãÔÐî×™ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.896' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0䊂ö‚(: —ðãÔÇ´°œ disk.nvme0n1.in‚ 710.1015625 disk.rbd0.in‚0& disk.nvme0c0n1.out‚39546.310546875 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 710.1015625$ memory_percent‚99.22025203704834( proc.memory.availableMB‚ 127.75390625 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2895.2578125 network.sent‚ 6.997351e+06! disk.nvme0c0n1.in‚ 710.1015625 disk.sda.in‚0 disk.sda.out‚ 108.11328125! disk.dm-0.out‚39546.310546875) proc.memory.percent‚17.67125129699707 disk.drbd1777.out‚0 cpu‚25.307338911932195( disk./.usagePercent‚72.7541009437298 disk.dm-1.out‚ 108.11328125 disk.sda3.in‚0 disk.sda3.out‚ 108.11328125 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0$ disk./.usageGB‚1084.088680267334 disk.rbd2.out‚58.63330078125 disk.rbd1.out‚1.4375 network.recv‚936328 disk.rbd3.out‚0$ disk.nvme0n1.out‚39546.310546875 proc.cpu.threads‚17›ŠT)2÷- _wandbruntime‚2496  _runtime‚2496ïc¨0ÇøÁ & lr-AdamW/pg1‚9.126817898203203e-05 & lr-AdamW/pg2‚9.126817898203203e-05 trainer/global_step‚3399 % _timestamp‚1.7884098864892576e+09 _runtime‚2496.901463851 _step‚274uÇÌùÆ & lr-AdamW/pg2‚9.126817898203203e-05 trainer/global_step‚3399 % _timestamp‚1.7884098864892576e+09 _runtime‚2496.901463851 _step‚274 & lr-AdamW/pg1‚9.126817898203203e-05’~W2ú- _wandbruntime‚2496  _runtime‚2496qºBÏ'ûÉO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 112831854 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44399077 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68432777 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19195043 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4864515003392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 % _timestamp‚1.7884098864900327e+09 _runtime‚2496.902459057 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4781109273088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚95105516719104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44399079 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4781109273088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90472292668416 trainer/global_step‚3399 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚99886625992192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚99886625992192 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99886438989824 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68432777 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45104049 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25909006 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚95105498716160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25909004 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚521 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4864725588480 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚519 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210585088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44399079 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68452814 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68452814 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚95105498716160 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4780940273664 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45103528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4780940273664 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99886438989824 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68432777 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45104049 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25909006 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚95105498716160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25909004 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚521 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4864725588480 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚519 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210585088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44399079 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68452814 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68452814 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚95105498716160 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4780940273664 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45103528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4780940273664 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68432777 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19195043 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4864515003392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 % _timestamp‚1.7884098864900327e+09 _runtime‚2496.902459057 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4781109273088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚95105516719104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44399079 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4781109273088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0“2´_2ý- _wandbruntime‚2497  _runtime‚2497\{™€Ù&þÓM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚2497.560308548 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚95266789740953 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚99915101492736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚90498218811392 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112884365 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 112864300 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19199860 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4866148260864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 221057536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112864300 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45116241 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 % _timestamp‚1.7884098871475058e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 _step‚276 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚95132786083840 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚95132768080896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚100814199211008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4782333411840 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44411736 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25916381 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105680347471872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68472627 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244997632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45116769 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44411736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25916383 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚90498235850752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚95266971901921 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚99915288508416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45116241 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 % _timestamp‚1.7884098871475058e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 _step‚276 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚95132786083840 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚95132768080896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚100814199211008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4782333411840 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44411736 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25916381 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105680347471872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68472627 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244997632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45116769 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44411736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25916383 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚90498235850752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚95266971901921 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚99915288508416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112884365 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 112864300 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19199860 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4866148260864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 221057536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112864300 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚101319578549248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45032027 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45032029 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45743521 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69445516 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69445516 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69425479 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45742958 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26277856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114457506 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682240 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚101319765551616 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚563 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 _step‚279 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45032027 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4935379529728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45032029 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚96468966952960 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚91769451785216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4836653583425 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4850780595712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚561 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚101319765551616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19465104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚102230909733888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚101319578549248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4850611596288 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚107166289263616 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96606088329281 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 trainer/global_step‚3449 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚91769434745856 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚102230933673984 _runtime‚ 2532.46263695 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26277854 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69425479 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4850611596288 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114477545 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚107166525885952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 $ _timestamp‚1.788409922050051e+09 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360"ÔºòÖ'ŒÐO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚101319578549248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4850611596288 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚107166289263616 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96606088329281 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 trainer/global_step‚3449 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚91769434745856 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚102230933673984 _runtime‚ 2532.46263695 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26277854 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69425479 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4850611596288 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114477545 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚107166525885952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 $ _timestamp‚1.788409922050051e+09 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4935592211968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚101319578549248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45032027 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45032029 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45743521 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69445516 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69445516 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69425479 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45742958 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26277856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114457506 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682240 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚101319765551616 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚563 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 _step‚279 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45032027 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4935379529728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45032029 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚96468966952960 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚91769451785216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4836653583425 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4850780595712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚561 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚101319765551616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19465104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚102230909733888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0—)€Í{2- _wandbruntime‚2533  _runtime‚2533@ajÃÜ&ŽÖM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4852173747200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45044688 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45755651 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102259846622208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚102259822682112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114489952 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚280 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45044688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4838207869577 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69465329 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4852004734464 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19470420 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236609024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45044686 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _runtime‚2533.133813515 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚101348428067840 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69465329 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69445266 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚96496236317696 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69445266 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚96496236317696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚91795394967552 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚91795377928192 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 114510017 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26285231 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3449 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114510017 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760¢oxk¿ÛM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚96496236317696 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69445266 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚96496236317696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚91795394967552 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚91795377928192 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 114510017 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26285231 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3449 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114510017 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4852173747200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45044688 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45755651 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102259846622208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚102259822682112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114489952 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚280 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45044688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4838207869577 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69465329 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4852004734464 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19470420 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236609024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45044686 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _runtime‚2533.133813515 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚101348428067840 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69465329 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69445266 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360˜Ö¨>2- _wandbruntime‚2533  _runtime‚2533É=V‘Ð 2loss/SRqqt2/assignment_loss‚1.7370569936492863 0loss/FBt4/assignment_loss‚0.6516109928009731 .loss/FRt4/detection_loss‚2.017539326633726 _runtime‚2533.135482039 0loss/FRt3/assignment_loss‚2.6304414259062874 2loss/SRqqt1/assignment_loss‚2.9695474842320317 .loss/FRt2/detection_loss‚2.159551441669464 1loss/FBt3/assignment_loss‚0.18579877853021212 _step‚281 0loss/FRt2/assignment_loss‚2.9219340060619596 1loss/SRqqt2/detection_loss‚1.4503292873501779 .loss/FBt4/detection_loss‚3.211898018916448 trainer/global_step‚3449 % _timestamp‚1.7884099227233815e+09 2loss/SRqqt3/assignment_loss‚1.8095623347908258 0loss/FBt1/assignment_loss‚0.4180578212589252 /loss/FRt3/detection_loss‚2.2524944386548467 1loss/SRqqt3/detection_loss‚1.7476830258965492 .loss/FBt1/detection_loss‚3.645020796193017 .loss/FBt2/detection_loss‚4.277315864960353 .loss/FBt3/detection_loss‚1.601758978537114 %loss/total_loss‚55.13067089346023 epoch‚2 /loss/FRt4/assignment_loss‚3.444694150061834 1loss/SRqqt4/assignment_loss‚1.697291859707167 0loss/FBt2/assignment_loss‚0.7896830101238569 1loss/SRqqt1/detection_loss‚2.9651807676190916 1loss/SRqqt4/detection_loss‚2.0254657781401346 /loss/FRt1/assignment_loss‚5.072888624100458 .loss/FRt1/detection_loss‚3.447865687665485-BØ[’Õ /loss/FRt4/assignment_loss‚3.444694150061834 1loss/SRqqt4/assignment_loss‚1.697291859707167 0loss/FBt2/assignment_loss‚0.7896830101238569 1loss/SRqqt1/detection_loss‚2.9651807676190916 1loss/SRqqt4/detection_loss‚2.0254657781401346 /loss/FRt1/assignment_loss‚5.072888624100458 .loss/FRt1/detection_loss‚3.447865687665485 2loss/SRqqt2/assignment_loss‚1.7370569936492863 0loss/FBt4/assignment_loss‚0.6516109928009731 .loss/FRt4/detection_loss‚2.017539326633726 _runtime‚2533.135482039 0loss/FRt3/assignment_loss‚2.6304414259062874 2loss/SRqqt1/assignment_loss‚2.9695474842320317 .loss/FRt2/detection_loss‚2.159551441669464 1loss/FBt3/assignment_loss‚0.18579877853021212 _step‚281 0loss/FRt2/assignment_loss‚2.9219340060619596 1loss/SRqqt2/detection_loss‚1.4503292873501779 .loss/FBt4/detection_loss‚3.211898018916448 trainer/global_step‚3449 % _timestamp‚1.7884099227233815e+09 2loss/SRqqt3/assignment_loss‚1.8095623347908258 0loss/FBt1/assignment_loss‚0.4180578212589252 /loss/FRt3/detection_loss‚2.2524944386548467 1loss/SRqqt3/detection_loss‚1.7476830258965492 .loss/FBt1/detection_loss‚3.645020796193017 .loss/FBt2/detection_loss‚4.277315864960353 .loss/FBt3/detection_loss‚1.601758978537114 %loss/total_loss‚55.13067089346023 epoch‚2™t;nÄ‘“‚(:† ÄðãÔò†Ú™ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.357' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6714 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lJ[L”‚(:ù ÄðãÔ̼° disk.rbd2.in‚0# memory_percent‚99.2210865020752$ disk.nvme0n1.out‚39745.396484375 network.recv‚952107 disk.rbd3.out‚0 disk.sda.out‚110 disk.drbd1777.out‚0! disk.dm-0.out‚39745.396484375 disk.rbd2.out‚ 58.8759765625 disk.sda2.out‚0 disk.sda.in‚0 disk.rbd1.out‚ 1.4697265625 disk.dm-0.in‚ 710.1015625' proc.memory.availableMB‚ 127.6171875 network.sent‚ 7.11753e+06% disk./.usageGB‚1084.0886993408203 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 710.1015625 disk.sda1.out‚0) proc.memory.percent‚17.67125129699707) disk./.usagePercent‚72.75410222376746 disk.dm-1.out‚110 disk.rbd0.in‚0 disk.sda3.in‚0& disk.nvme0c0n1.out‚39745.396484375 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda3.out‚110 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 710.1015625" proc.memory.rssMB‚ 2895.2578125 cpu‚25.302989761433214 proc.cpu.threads‚17óÌ$Α•‚(:† ÓðãÔ÷‹“™ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.756' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T$Z –‚(:• ÓðãÔ¬‡œ network.recv‚958057 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd2.out‚ 58.8759765625 disk.rbd3.in‚0 disk.sda.out‚ 110.4453125 disk.rbd1.out‚ 1.53271484375! disk.dm-0.out‚39844.478515625 disk.dm-1.in‚0 disk.drbd1777.out‚0 network.sent‚ 7.225572e+06 disk.dm-1.out‚ 110.4453125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚ 710.1015625$ disk.nvme0n1.out‚39844.478515625) disk./.usagePercent‚72.75410222376746 disk.dm-0.in‚ 710.1015625! disk.nvme0c0n1.in‚ 710.1015625$ memory_percent‚99.21934604644775) proc.memory.percent‚17.67125129699707% disk./.usageGB‚1084.0886993408203 disk.sda3.out‚ 110.4453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 127.90234375 disk.sda3.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚39844.478515625" proc.memory.rssMB‚ 2895.2578125 cpu‚25.301445637894094I8Š6‘—‚(:† âðãÔå‚«™ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.584' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5168 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OCôU"˜‚(:— âðãÔ°Ô­› disk.drbd1777.in‚0 disk.rbd1.out‚ 1.53271484375 disk.dm-1.out‚ 111.0859375 disk.rbd2.out‚ 60.5283203125 disk.rbd3.out‚0 disk.sda.in‚0 disk.nvme0n1.in‚ 710.1015625" disk.dm-0.out‚39844.9267578125 disk.sda2.out‚0 disk.sda3.in‚0$ memory_percent‚99.21936988830566 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0' proc.memory.availableMB‚ 127.8984375" proc.memory.rssMB‚ 2895.2578125 network.sent‚ 7.231818e+06) disk./.usagePercent‚72.75410043171472 disk.rbd0.out‚0 disk.sda.out‚ 111.0859375 disk.dm-0.in‚ 710.1015625! disk.nvme0c0n1.in‚ 710.1015625 cpu‚25.300003276560282 proc.cpu.threads‚17% disk./.usageGB‚1084.0886726379395 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.out‚0% disk.nvme0n1.out‚39844.9267578125' disk.nvme0c0n1.out‚39844.9267578125 disk.sda3.out‚ 111.0859375) proc.memory.percent‚17.67125129699707 network.recv‚962856û¶U 2™- _wandbruntime‚2567  _runtime‚2567Â\[ÇšÁ & lr-AdamW/pg2‚9.038930468709887e-05 trainer/global_step‚3499 % _timestamp‚1.7884099574158168e+09 _runtime‚2567.827993824 _step‚282 & lr-AdamW/pg1‚9.038930468709887e-05`zÆäÌ›Æ _step‚282 & lr-AdamW/pg1‚9.038930468709887e-05 & lr-AdamW/pg2‚9.038930468709887e-05 trainer/global_step‚3499 % _timestamp‚1.7884099574158168e+09 _runtime‚2567.827993824šg/þÕ2œ- _wandbruntime‚2567  _runtime‚2567R“°’Ó'ÍO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4906290523233 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 _step‚283 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4920282918912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚97972901425249 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70418181 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108682814532096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚97832453192704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚97832435189760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚97832453192704 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26646704 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚102752905111040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70438218 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚102752905111040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116103197 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚103676557145088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚3499 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46385687 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108683046960128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4920451918336 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚102752718108672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5006465874944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4906125401833 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚102752718108672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116083158 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70438218 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26646706 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46386228 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0Ô8ÈžÒO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚103676557145088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚3499 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46385687 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108683046960128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4920451918336 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚102752718108672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5006465874944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4906125401833 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚102752718108672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116083158 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70438218 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26646706 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46386228 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4906290523233 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 _step‚283 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4920282918912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚97972901425249 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70418181 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108682814532096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚97832453192704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚97832435189760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚97832453192704 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26646704 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚102752905111040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70438218 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚102752905111040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116103197 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45677638 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _step‚284 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4921845069824 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚93092537044992 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4907679687985 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚102781380611584 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚2568.504511611 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚97859704554496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70458031 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108713378629120 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108713146214400 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4921845069824 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚98000051611369 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚93092554084352 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116115604 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116135669 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚98000233772337 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46398425 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚3499 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4921676057088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116135669 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70437968 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116115604 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚102781567627264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45677638 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46398910 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19744344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192&4Áñâ&¡ÜM F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108713378629120 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108713146214400 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4921845069824 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚98000051611369 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚93092554084352 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116115604 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116135669 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚98000233772337 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46398425 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚3499 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4921676057088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116135669 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70437968 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116115604 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚102781567627264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45677638 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46398910 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19744344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7884099580912344e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5007676121088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4921676057088 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚103705470093312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚102781567627264 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70437968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45677638 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _step‚284 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4921845069824 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚93092537044992 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4907679687985 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚102781380611584 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚2568.504511611 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚97859704554496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70458031œ磺2¢- _wandbruntime‚2568  _runtime‚2568»íx€Z£Ô 1loss/SRqqt2/detection_loss‚1.5004325063831836 %loss/total_loss‚51.83704356204884 2loss/SRqqt2/assignment_loss‚1.9736833561615772 /loss/FRt2/detection_loss‚2.2589343167700857 trainer/global_step‚3499 _runtime‚2568.505777346 0loss/FRt2/assignment_loss‚3.2129458953749457 /loss/FRt4/assignment_loss‚4.252871873586074 2loss/SRqqt4/assignment_loss‚2.0046040073354194 /loss/FBt3/detection_loss‚0.9857081735186228 $ _timestamp‚1.788409958093709e+09 _step‚285 /loss/FRt1/assignment_loss‚4.675415867253354 2loss/SRqqt1/assignment_loss‚2.0098132819964967 0loss/FBt2/assignment_loss‚1.0074637432893117 0loss/FBt1/assignment_loss‚0.5876346602053673 0loss/FBt3/assignment_loss‚0.2789376433826912 /loss/FRt4/detection_loss‚2.2686111933511235 1loss/SRqqt4/detection_loss‚1.9669579907934716 .loss/FBt1/detection_loss‚3.047157875129155 /loss/FBt2/detection_loss‚1.8038750290870667 /loss/FBt4/detection_loss‚2.7251851558685303 0loss/SRqqt3/detection_loss‚1.962399990260601 0loss/FRt3/assignment_loss‚2.7613213500886595 0loss/FBt4/assignment_loss‚0.6637502494461212 2loss/SRqqt3/assignment_loss‚1.6266076311469078 /loss/FRt1/detection_loss‚3.4827618285229334 /loss/FRt3/detection_loss‚2.0421399249220795 epoch‚2 1loss/SRqqt1/detection_loss‚2.7378300181750594¥ßÚ“_¤Ù 0loss/FBt4/assignment_loss‚0.6637502494461212 2loss/SRqqt3/assignment_loss‚1.6266076311469078 /loss/FRt1/detection_loss‚3.4827618285229334 /loss/FRt3/detection_loss‚2.0421399249220795 epoch‚2 1loss/SRqqt1/detection_loss‚2.7378300181750594 1loss/SRqqt2/detection_loss‚1.5004325063831836 %loss/total_loss‚51.83704356204884 2loss/SRqqt2/assignment_loss‚1.9736833561615772 /loss/FRt2/detection_loss‚2.2589343167700857 trainer/global_step‚3499 _runtime‚2568.505777346 0loss/FRt2/assignment_loss‚3.2129458953749457 /loss/FRt4/assignment_loss‚4.252871873586074 2loss/SRqqt4/assignment_loss‚2.0046040073354194 /loss/FBt3/detection_loss‚0.9857081735186228 $ _timestamp‚1.788409958093709e+09 _step‚285 /loss/FRt1/assignment_loss‚4.675415867253354 2loss/SRqqt1/assignment_loss‚2.0098132819964967 0loss/FBt2/assignment_loss‚1.0074637432893117 0loss/FBt1/assignment_loss‚0.5876346602053673 0loss/FBt3/assignment_loss‚0.2789376433826912 /loss/FRt4/detection_loss‚2.2686111933511235 1loss/SRqqt4/detection_loss‚1.9669579907934716 .loss/FBt1/detection_loss‚3.047157875129155 /loss/FBt2/detection_loss‚1.8038750290870667 /loss/FBt4/detection_loss‚2.7251851558685303 0loss/SRqqt3/detection_loss‚1.962399990260601 0loss/FRt3/assignment_loss‚2.7613213500886595%k¾Á¥‚(:… ñðãÔ’Êœ™ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.495' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.299 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N¼ég¦‚(:” ñðãÔÄÏæ› disk.sda2.in‚0 disk.sda1.out‚0$ memory_percent‚99.22330379486084( proc.memory.availableMB‚ 127.25390625 proc.cpu.threads‚17 disk.rbd2.out‚ 60.5283203125! disk.nvme0c0n1.in‚ 710.1015625 disk.nvme0n1.in‚ 710.1015625 disk.dm-1.in‚0" proc.memory.rssMB‚ 2895.2578125 disk.sda.in‚0 disk.sda.out‚ 111.66015625! disk.dm-0.out‚39944.064453125& disk.nvme0c0n1.out‚39944.064453125 disk.rbd0.in‚0 disk.rbd2.in‚0$ disk.nvme0n1.out‚39944.064453125( disk./.usagePercent‚72.7540942875339 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.595703125) proc.memory.percent‚17.67125129699707 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 111.66015625 disk.dm-0.in‚ 710.1015625$ disk./.usageGB‚1084.088581085205 disk.rbd1.in‚0 disk.dm-1.out‚ 111.66015625 cpu‚25.298650425740963 network.sent‚ 7.237983e+06 disk.rbd0.out‚0 disk.drbd1777.in‚0 network.recv‚967465cÅÍhš§‚(: €ñãÔј§™ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.751' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.750200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'‰T #¨‚(:˜ €ñãÔÜœ£œ disk.drbd1777.out‚0 disk.rbd1.in‚0 cpu‚25.297217378811535 disk.dm-1.out‚ 112.08203125 disk.rbd3.in‚0 disk.sda.out‚ 112.08203125 disk.drbd1777.in‚0 network.sent‚ 7.345096e+06& disk.nvme0n1.out‚40043.31103515625 disk.rbd0.out‚0 disk.rbd2.out‚ 60.6689453125! disk.nvme0c0n1.in‚ 710.1015625( disk.nvme0c0n1.out‚40043.31103515625 disk.sda3.in‚0$ memory_percent‚99.22189712524414 network.recv‚972738 disk.rbd0.in‚0& proc.memory.availableMB‚ 127.484375 disk.nvme0n1.in‚ 710.1015625 disk.dm-0.in‚ 710.1015625 disk.sda3.out‚ 112.08203125 disk.rbd1.out‚ 1.595703125" proc.memory.rssMB‚ 2895.2578125) proc.memory.percent‚17.67125129699707 proc.cpu.threads‚17# disk.dm-0.out‚40043.31103515625 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0( disk./.usagePercent‚72.7540942875339$ disk./.usageGB‚1084.088581085205 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$J=ª2©- _wandbruntime‚2603  _runtime‚2603ï‘{ǪÁ trainer/global_step‚3549 % _timestamp‚1.7884099930039964e+09 _runtime‚2603.416168807 _step‚286 & lr-AdamW/pg1‚8.993735744123257e-05 & lr-AdamW/pg2‚8.993735744123257e-05xFgÌ«Æ _runtime‚2603.416168807 _step‚286 & lr-AdamW/pg1‚8.993735744123257e-05 & lr-AdamW/pg2‚8.993735744123257e-05 trainer/global_step‚3549 % _timestamp‚1.7884099930039964e+09ž¸Ó–u2¬- _wandbruntime‚2603  _runtime‚2603DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚104185857668096 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚99195921429504 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71430920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210585088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4975597220241 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5077157386752 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚99339350199697 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46297929 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47027307 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4975762341641 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚104186044670464 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚110199596468224 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27015554 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46297927 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚3549 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚99195903426560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚99195903426560 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4990123240960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234525184 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚104186044670464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚104185857668096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚287 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46297927 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚105122204556288 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚94363752979456 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4989954241536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46297929 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚570 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27015556 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚99339532360457 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5077367971840 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚104185857668096 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚99195921429504 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71430920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210585088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4975597220241 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5077157386752 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚99339350199697 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46297929 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47027307 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4975762341641 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚104186044670464 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚110199596468224 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27015554 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46297927 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚3549 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚99195903426560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚99195903426560 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4990123240960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234525184 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚104186044670464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚104185857668096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚287 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46297927 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚105122204556288 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚94363752979456 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4989954241536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46297929 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚570 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27015556 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚99339532360457 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5077367971840 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47040015 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200086016 _step‚288 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46310588 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117761321 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27022933 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 2604.07156549 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46310586 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚99223190794240 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚105151117504512 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224026112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚99366682546577 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚99223190794240 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚99223172791296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚504 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117741256 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4991516392448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚94389713201152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4977151506393 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚110229702119424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚99223172791296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71450733 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚99223190794240 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚105151117504512 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224026112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚99366682546577 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚99223190794240 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚99223172791296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚504 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117741256 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 SD1OçÊzDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4991516392448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚94389713201152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4977151506393 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚110229702119424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚99223172791296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71450733 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47040015 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200086016 _step‚288 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46310588 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117761321 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27022933 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ¿Ü¦2²- _wandbruntime‚2604  _runtime‚2604$,T³Î %loss/total_loss‚64.17777258959146 epoch‚2 .loss/FRt3/detection_loss‚2.156881525814533 /loss/FRt1/assignment_loss‚5.190519762039185 0loss/SRqqt4/detection_loss‚2.064871909136468 /loss/FBt3/detection_loss‚0.8861042032807332 _runtime‚2604.073074874 0loss/FRt3/assignment_loss‚2.8767044854164125 .loss/FRt2/detection_loss‚2.366239853776418 /loss/FRt4/assignment_loss‚3.905937728782495 0loss/FBt3/assignment_loss‚0.2592582252486303 0loss/FBt4/assignment_loss‚0.6384058490395546 /loss/FRt4/detection_loss‚2.1472397899876037 1loss/SRqqt2/detection_loss‚1.7706222037474315 /loss/FBt1/detection_loss‚3.5075184161012825 1loss/SRqqt1/assignment_loss‚2.276097826957703 /loss/FRt1/detection_loss‚2.9578534507751466 trainer/global_step‚3549 1loss/SRqqt2/assignment_loss‚1.804780567901722 2loss/SRqqt4/assignment_loss‚1.7172041342772069 0loss/FBt1/assignment_loss‚0.5172952048120681 0loss/SRqqt3/detection_loss‚1.682833173987912 .loss/FBt4/detection_loss‚1.927915451924006 % _timestamp‚1.7884099936610184e+09 _step‚289 /loss/FRt2/assignment_loss‚4.198907110553521 2loss/SRqqt3/assignment_loss‚2.0558573841756465 0loss/FBt2/assignment_loss‚1.3487505912780762 0loss/SRqqt1/detection_loss‚2.820057759284973 /loss/FBt2/detection_loss‚13.099915981292725Ò¦VpY´Ó 0loss/FBt2/assignment_loss‚1.3487505912780762 0loss/SRqqt1/detection_loss‚2.820057759284973 /loss/FBt2/detection_loss‚13.099915981292725 %loss/total_loss‚64.17777258959146 epoch‚2 .loss/FRt3/detection_loss‚2.156881525814533 /loss/FRt1/assignment_loss‚5.190519762039185 0loss/SRqqt4/detection_loss‚2.064871909136468 /loss/FBt3/detection_loss‚0.8861042032807332 _runtime‚2604.073074874 0loss/FRt3/assignment_loss‚2.8767044854164125 .loss/FRt2/detection_loss‚2.366239853776418 /loss/FRt4/assignment_loss‚3.905937728782495 0loss/FBt3/assignment_loss‚0.2592582252486303 0loss/FBt4/assignment_loss‚0.6384058490395546 /loss/FRt4/detection_loss‚2.1472397899876037 1loss/SRqqt2/detection_loss‚1.7706222037474315 /loss/FBt1/detection_loss‚3.5075184161012825 1loss/SRqqt1/assignment_loss‚2.276097826957703 /loss/FRt1/detection_loss‚2.9578534507751466 trainer/global_step‚3549 1loss/SRqqt2/assignment_loss‚1.804780567901722 2loss/SRqqt4/assignment_loss‚1.7172041342772069 0loss/FBt1/assignment_loss‚0.5172952048120681 0loss/SRqqt3/detection_loss‚1.682833173987912 .loss/FBt4/detection_loss‚1.927915451924006 % _timestamp‚1.7884099936610184e+09 _step‚289 /loss/FRt2/assignment_loss‚4.198907110553521 2loss/SRqqt3/assignment_loss‚2.0558573841756465¡º–@µ‚(:… ñãÔ Á±™ gpu.0.gpu‚21.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚89.613' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.9226 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 gÔ³#¶‚(:˜ ñãÔ¤ÂÍœ disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 710.1015625 disk.rbd2.out‚ 60.6689453125 disk.sda2.out‚0 disk.sda3.out‚ 112.7578125 cpu‚25.295803239517063 disk.dm-1.out‚ 112.7578125 disk.rbd3.out‚0" proc.memory.rssMB‚ 2895.2578125 disk.sda3.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd1.out‚ 1.595703125 disk.nvme0n1.in‚ 710.1015625) proc.memory.percent‚17.67125129699707 disk.dm-1.in‚0( disk.nvme0c0n1.out‚40043.61181640625 disk.sda.out‚ 112.7578125 disk.drbd1777.in‚0 disk.rbd1.in‚0# disk.dm-0.out‚40043.61181640625$ disk./.usageGB‚1084.089038848877 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 710.1015625$ memory_percent‚99.22349452972412( proc.memory.availableMB‚ 127.22265625 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.drbd1777.out‚0& disk.nvme0n1.out‚40043.61181640625 network.sent‚ 7.350615e+06 network.recv‚976276) disk./.usagePercent‚72.75412500843801Q݆¤›·‚(: žñãÔ›¹¶™ gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚98.472' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.694399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00‚¤'¸‚(:œ žñãÔú„Ûœ proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.595703125 disk.rbd0.in‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚40143.70556640625 disk.sda2.out‚0& disk.nvme0n1.out‚40143.70556640625) disk./.usagePercent‚72.73882190207127 disk.sda.in‚0 disk.sda.out‚ 113.32421875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.dm-1.out‚ 113.32421875 disk.rbd2.out‚60.77099609375 disk.sda3.out‚ 113.32421875) proc.memory.percent‚17.67125129699707 cpu‚25.294409660718156$ disk./.usageGB‚1083.861011505127 disk.rbd1.in‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 127.17578125 disk.nvme0n1.in‚ 710.1015625 disk.dm-1.in‚0 disk.sda3.in‚0$ memory_percent‚99.22378063201904! disk.nvme0c0n1.in‚ 710.1015625 network.sent‚ 7.459534e+06 disk.sda1.out‚0 disk.rbd2.in‚0# disk.dm-0.out‚40143.70556640625" proc.memory.rssMB‚ 2895.2578125 network.recv‚983623 disk.dm-0.in‚ 710.1015625 disk.sda2.in‚0‰Ç2¹- _wandbruntime‚2638  _runtime‚2638Õÿ…ƺÀ & lr-AdamW/pg1‚8.947719513272635e-05 & lr-AdamW/pg2‚8.947719513272635e-05 trainer/global_step‚3599 $ _timestamp‚1.788410028470901e+09 _runtime‚2638.883119216 _step‚290Ö;Z˻Š_runtime‚2638.883119216 _step‚290 & lr-AdamW/pg1‚8.947719513272635e-05 & lr-AdamW/pg2‚8.947719513272635e-05 trainer/global_step‚3599 $ _timestamp‚1.788410028470901e+09¢œº2¼- _wandbruntime‚2638  _runtime‚2638l. ×'½ÑO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚100559389666304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20283270 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105618997227520 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20282744 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46930877 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5148188323840 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚100705981134905 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 $ _timestamp‚1.788410028471651e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 119354501 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95660912096256 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5059794563584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚528 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚106567851967488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 119334462 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72423622 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5045069038649 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5059625564160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46930879 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95660929135616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚100559389666304 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105618997227520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47667148 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487936 _runtime‚ 2638.88400794 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚526 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5059625564160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46930877 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27384406 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚100706163295665 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105619184229888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72403585 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46930879 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚105619184229888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚111716040291328 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47667676 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5148396811776 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248Ñ)ÓøÜ'¾ÖO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5059794563584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95660912096256 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5059794563584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚528 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚106567851967488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 119334462 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72423622 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5045069038649 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5059625564160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46930879 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95660929135616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚100559389666304 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105618997227520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47667148 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487936 _runtime‚ 2638.88400794 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚526 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5059625564160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46930877 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27384406 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚100706163295665 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105619184229888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72403585 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46930879 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚105619184229888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚111716040291328 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47667676 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5148396811776 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚100559389666304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20283270 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105618997227520 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20282744 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46930877 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5148188323840 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚100705981134905 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 $ _timestamp‚1.788410028471651e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 119354501 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72423372 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚100586641028096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46943538 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27391783 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20288434 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚106596788855808 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5061018702336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5061018702336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚105647846746112 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5046623324801 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚2639.554304457 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚105647659730432 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72443435 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚100733313481785 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚95686872317952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95686855278592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119386973 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚111746376683008 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚100586659031040 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46943536 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5149816047616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 $ _timestamp‚1.788410029141359e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119366908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27391781 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚106596764915712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚100586641028096 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72443435 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚510 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47680215 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚100733495642753 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72423372 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204280320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5061187715072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚105647846746112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚111746604903424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚508 _step‚292 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2Æ_>5ç&ÁáM _runtime‚2639.554304457 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚105647659730432 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72443435 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚100733313481785 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚95686872317952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95686855278592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119386973 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚111746376683008 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚100586659031040 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46943536 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5149816047616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 $ _timestamp‚1.788410029141359e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119366908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27391781 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚106596764915712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚100586641028096 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72443435 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚510 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47680215 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚100733495642753 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72423372 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204280320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5061187715072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚105647846746112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚111746604903424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚508 _step‚292 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119386973 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72423372 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚100586641028096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46943538 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27391783 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20288434 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚106596788855808 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5061018702336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5061018702336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚105647846746112 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5046623324801 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160¤ ÒC2Â- _wandbruntime‚2639  _runtime‚2639nò ¯YÃÓ 0loss/FRt4/assignment_loss‚3.8997358944680958 .loss/FRt2/detection_loss‚2.357735182179345 /loss/FRt4/detection_loss‚2.2856010033024683 .loss/FBt2/detection_loss‚3.148677852418688 &loss/total_loss‚53.569422793038996 % _timestamp‚1.7884100291431792e+09 2loss/SRqqt1/assignment_loss‚2.5307577773928642 2loss/SRqqt3/assignment_loss‚1.8543557540366524 /loss/FBt1/detection_loss‚3.8067098723517523 trainer/global_step‚3599 /loss/FRt1/assignment_loss‚4.283997740064349 0loss/FRt3/assignment_loss‚2.2227427592644324 0loss/FBt1/assignment_loss‚0.2612295581172172 1loss/FBt3/assignment_loss‚0.14946384926073925 1loss/SRqqt3/detection_loss‚1.5578907827536266 1loss/SRqqt4/detection_loss‚2.1044069570302963 -loss/FBt4/detection_loss‚4.36476335922877 epoch‚2 /loss/FRt2/assignment_loss‚4.111328629652659 2loss/SRqqt2/assignment_loss‚1.7117749081572164 0loss/FBt2/assignment_loss‚0.8123918606175317 .loss/FRt1/detection_loss‚3.031158208847046 1loss/SRqqt1/detection_loss‚2.4549997663125396 1loss/FBt4/assignment_loss‚0.42554371306884303 1loss/SRqqt2/detection_loss‚1.4849628862496969 /loss/FBt3/detection_loss‚0.8589693902524104 _step‚293 2loss/SRqqt4/assignment_loss‚1.9899723204970359 .loss/FRt3/detection_loss‚1.860252767514724 _runtime‚ 2639.55537809xœÝ¯^ÄØ 0loss/FRt4/assignment_loss‚3.8997358944680958 .loss/FRt2/detection_loss‚2.357735182179345 /loss/FRt4/detection_loss‚2.2856010033024683 .loss/FBt2/detection_loss‚3.148677852418688 &loss/total_loss‚53.569422793038996 % _timestamp‚1.7884100291431792e+09 2loss/SRqqt1/assignment_loss‚2.5307577773928642 2loss/SRqqt3/assignment_loss‚1.8543557540366524 /loss/FBt1/detection_loss‚3.8067098723517523 trainer/global_step‚3599 /loss/FRt1/assignment_loss‚4.283997740064349 0loss/FRt3/assignment_loss‚2.2227427592644324 0loss/FBt1/assignment_loss‚0.2612295581172172 1loss/FBt3/assignment_loss‚0.14946384926073925 1loss/SRqqt3/detection_loss‚1.5578907827536266 1loss/SRqqt4/detection_loss‚2.1044069570302963 -loss/FBt4/detection_loss‚4.36476335922877 epoch‚2 /loss/FRt2/assignment_loss‚4.111328629652659 2loss/SRqqt2/assignment_loss‚1.7117749081572164 0loss/FBt2/assignment_loss‚0.8123918606175317 .loss/FRt1/detection_loss‚3.031158208847046 1loss/SRqqt1/detection_loss‚2.4549997663125396 1loss/FBt4/assignment_loss‚0.42554371306884303 1loss/SRqqt2/detection_loss‚1.4849628862496969 /loss/FBt3/detection_loss‚0.8589693902524104 _step‚293 2loss/SRqqt4/assignment_loss‚1.9899723204970359 .loss/FRt3/detection_loss‚1.860252767514724 _runtime‚ 2639.55537809¥É}Ï™œÅ‚(:‘ ­ñãÔüÄÑ™ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚101.613' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.322599999999998 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|ñ˜%Æ‚(:š ­ñãÔ¸ÚÓœ# disk.dm-0.out‚41306.37060546875 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 710.1015625 cpu‚25.293128516900882 proc.cpu.threads‚17 network.sent‚ 7.4656e+06! disk.nvme0c0n1.in‚ 710.1015625& disk.nvme0n1.out‚41306.37060546875 disk.dm-0.in‚ 710.1015625 disk.rbd3.in‚0 disk.sda3.in‚0 disk.sda.out‚ 114.88671875' proc.memory.availableMB‚ 127.2734375 disk.dm-1.in‚0 disk.dm-1.out‚ 114.88671875 disk.rbd0.out‚0 disk.sda3.out‚ 114.88671875" proc.memory.rssMB‚ 2895.2578125 network.recv‚988116) disk./.usagePercent‚72.90722724220001 disk.rbd2.out‚60.77099609375) proc.memory.percent‚17.67125129699707 disk.rbd3.out‚0( disk.nvme0c0n1.out‚41306.37060546875 disk.sda.in‚0 disk.rbd1.out‚ 1.595703125% disk./.usageGB‚1086.3703727722168 disk.sda2.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0$ memory_percent‚99.22318458557129 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda1.in‚0–b‘Ç‚(:† ¼ñãÔ«¬±™ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.687' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1374 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆj$È‚(:™ ¼ñãÔËÍû› network.recv‚994452% disk./.usageGB‚1088.3528518676758 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 115.51171875" proc.memory.rssMB‚ 2895.2578125) disk./.usagePercent‚73.04027307770701 disk.nvme0n1.in‚ 710.1015625 disk.rbd2.in‚0 disk.rbd2.out‚62.48583984375 disk.rbd3.in‚0$ memory_percent‚99.22471046447754 disk.drbd1777.out‚0 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 710.1015625 disk.sda.out‚ 115.51171875 cpu‚25.291781384101654 disk.drbd1777.in‚0 disk.dm-1.in‚0' proc.memory.availableMB‚ 127.0234375% disk.nvme0n1.out‚41346.0068359375) proc.memory.percent‚17.67125129699707 proc.cpu.threads‚17 disk.sda1.in‚0' disk.nvme0c0n1.out‚41346.0068359375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 710.1015625" disk.dm-0.out‚41346.0068359375 disk.dm-1.out‚ 115.51171875 network.sent‚ 7.573508e+06 disk.rbd1.out‚ 1.595703125 disk.rbd0.in‚0 $‚‡É‚(:… ËñãÔò„ý˜ gpu.0.gpu‚43.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.825' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.565 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬RTÀÊ‚(:Œ ËñãÔÑ“¿œ disk.sda2.in‚0 disk.sda3.out‚ 116.640625 disk.sda1.out‚0 disk.rbd1.in‚0" proc.memory.rssMB‚ 2895.2578125 network.sent‚ 7.579008e+06 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 710.31640625) proc.memory.percent‚17.67125129699707 cpu‚25.314083231180938 disk.rbd3.in‚0' proc.memory.availableMB‚ 127.0234375 disk.sda3.in‚0 disk.dm-0.in‚ 710.31640625 disk.dm-1.out‚ 116.640625 disk.rbd0.out‚0$ disk.nvme0c0n1.out‚ 42531.1953125) disk./.usagePercent‚73.02265028707251 disk.sda.out‚ 116.640625" disk.nvme0n1.out‚ 42531.1953125 disk.rbd2.out‚62.48583984375 disk.rbd3.out‚0$ disk./.usageGB‚1088.090259552002$ memory_percent‚99.22471046447754 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.dm-0.out‚ 42531.1953125 disk.dm-1.in‚0 disk.rbd2.in‚0 network.recv‚998022 disk.sda2.out‚0 disk.sda.in‚0 disk.rbd1.out‚ 1.595703125 disk.nvme0n1.in‚ 710.31640625ukò0Ë‚(:„ ÚñãÔëŽ×™ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹Š²zÌ‚(:“ ÚñãÔìÙùœ disk.rbd1.out‚ 1.595703125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.drbd1777.out‚0' proc.memory.availableMB‚ 127.2734375 proc.cpu.threads‚17 network.recv‚ 1.002878e+06 disk.dm-1.out‚ 116.98046875 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 62.650390625 disk.sda.out‚ 116.98046875 network.sent‚ 7.585278e+06 disk.sda3.out‚ 116.98046875 disk.sda1.in‚0" proc.memory.rssMB‚ 2895.2578125 cpu‚25.33955334275032 disk.nvme0n1.in‚ 710.40625$ disk.nvme0n1.out‚45450.802734375 disk.rbd3.in‚0 disk.sda.in‚0! disk.dm-0.out‚45450.802734375 disk.sda3.in‚0) proc.memory.percent‚17.67125129699707% disk./.usageGB‚1089.3444557189941 disk.rbd1.in‚0 disk.dm-1.in‚0& disk.nvme0c0n1.out‚45450.802734375) disk./.usagePercent‚73.10682044418003 disk.dm-0.in‚ 710.40625 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.drbd1777.in‚0$ memory_percent‚99.22318458557129åx"‘Í‚(:† éñãÔšÒ±š gpu.0.gpu‚37.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.398' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤ý΂(: éñãÔ í¿Ÿ$ disk.nvme0n1.out‚45881.455078125 disk.dm-0.in‚ 710.40625! disk.dm-0.out‚45881.455078125 disk.sda1.out‚0( disk./.usagePercent‚73.0916348452715 disk.drbd1777.out‚0 disk.dm-1.out‚ 118.01171875 disk.nvme0c0n1.in‚ 710.40625& disk.nvme0c0n1.out‚45881.455078125 disk.sda2.out‚0 disk.rbd1.out‚ 1.595703125 disk.nvme0n1.in‚ 710.40625 disk.rbd3.in‚0% proc.memory.availableMB‚ 127.28125 network.sent‚ 7.593495e+06 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda.out‚ 118.01171875 cpu‚25.36910615135875 proc.cpu.threads‚17) proc.memory.percent‚17.67125129699707 network.recv‚ 1.007102e+06 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 62.650390625 disk.sda3.out‚ 118.01171875 disk.sda1.in‚0$ disk./.usageGB‚1089.118179321289$ memory_percent‚99.22313690185547" proc.memory.rssMB‚ 2895.2578125'Kg—2Ï- _wandbruntime‚2706  _runtime‚2706 ù@йT TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 121150754 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2460 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚199 % _timestamp‚1.7884100956153755e+09 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24400 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚73324559 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚199 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚47810219 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚47810217 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚107363255869440 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚107363110558720 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1003 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24127 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 5115929886720 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚416 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15976 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 4912020480 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚113543170357248 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 140964948 _runtime‚ 2706.02807366 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 410711552 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚416 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 410711552 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 121134776 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚108338013170688 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 5205181126656 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚102365698751501 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚27909462 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚281 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 5101060455353 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 417333248 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15978 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚283 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.alloëz'cated‚97264514370560 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 5041553408 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 5115802578944 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚20672576 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 145310720 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚102247325982720 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 123925588 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 121134776 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚73340535 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 131513344 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚73340535 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 5115929886720 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 127307776 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 121150754 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚113543301870592 trainer/global_step‚3626 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 140964948 _runtime‚ 2706.02807366 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 410711552 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚416 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 410711552 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 121134776 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚108338013170688 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 5205181126656 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚102365698751501 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚27909462 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚281 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 5101060455353 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 417333248 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15978 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚283 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚97264514370560 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 5041553408 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 5115802578944 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚20672576 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 145310720 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚102247325982720 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 123925588 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 121134776 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚73340535 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 131513344 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚73340535 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 5115929886720 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 127307776 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 121150754 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚113543301870592 trainer/global_step‚3626 FDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24400 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚48593457 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚20677214 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 5041553408 % _timestamp‚1.7884100965336435e+09 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚108357592796160 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 4624220160 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2460 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1003 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚27915962 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 131513344 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 145310720 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚107382324880384 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚283 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚217 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚102266191168512 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚217 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 5205530262528 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 _runtime‚2706.946410027 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚113563254572032 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 107573248 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚97282398884864 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 417333248 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 121168862 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 5116279022592 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚73347931 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 4757245253 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 5101408061689 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚107382470191104 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 127307776 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚199 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 121152884 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚73347931 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 5116151714816 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚102266173165568 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 4912020480 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 4541568000 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚47820931 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚416 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 121152884 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 140964948 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚102383806946553 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _step‚295 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚73331955 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 145310720 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24127 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 5116279022592 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚48593174 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚416 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 417333248 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15976 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 410711552 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 4912020480 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚102266173165568 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚102266191168512 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚47820929 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚102383947911501 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2460 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15976 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚73331955 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚26 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 406789965 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 127307776 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24127 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚108357616736256 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24127 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 5116279022592 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚48593174 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚416 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 417333248 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15976 UGDeviceStatsMonitor.on_validŠù[*Yation_batch_end/active_bytes.small_pool.peak‚ 410711552 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 4912020480 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚102266173165568 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚102266191168512 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚47820929 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚102383947911501 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2460 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15976 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚73331955 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚26 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 406789965 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 127307776 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24127 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚108357616736256 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24400 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚48593457 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚20677214 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 5041553408 % _timestamp‚1.7884100965336435e+09 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚108357592796160 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 4624220160 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2460 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1003 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚27915962 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 131513344 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 145310720 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚107382324880384 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚283 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚217 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚102266191168512 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚217 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 5205530262528 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 _runtime‚2706.946410027 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚113563254572032 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 107573248 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚97282398884864 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 417333248 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 121168862 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 5116279022592 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚73347931 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 4757245253 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 5101408061689 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚107382470191104 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 127307776 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚199 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 121152884 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚73347931 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 5116151714816 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚102266173165568 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 4912020480 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 4541568000 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0§”5ÐxÕ‚(:„ øñãÔÇÍå™ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.825' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.565 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q2û%Ö‚(:š øñãÔÛ´³œ network.recv‚ 1.010953e+06 disk.rbd1.out‚ 1.595703125 disk.dm-0.in‚ 710.40625' proc.memory.availableMB‚ 127.4765625) disk./.usagePercent‚73.09157468350097 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.rbd2.out‚ 62.787109375 network.sent‚ 7.596782e+06 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0( disk.nvme0c0n1.out‚45988.18798828125$ memory_percent‚99.22194480895996) proc.memory.percent‚17.67125129699707 proc.cpu.threads‚17 disk.sda.in‚0 disk.nvme0n1.in‚ 710.40625# disk.dm-0.out‚45988.18798828125 disk.dm-1.out‚ 118.53515625 disk.rbd0.out‚0 disk.sda3.out‚ 118.53515625 disk.sda1.out‚0 cpu‚25.398354287218606% disk./.usageGB‚1089.1172828674316 disk.drbd1777.out‚0 disk.rbd1.in‚0" proc.memory.rssMB‚ 2895.2578125 disk.nvme0c0n1.in‚ 710.40625 disk.sda3.in‚0 disk.sda.out‚ 118.53515625& disk.nvme0n1.out‚45988.18798828125 disk.sda2.in‚0 disk.sda1.in‚0+%ÞÄ2×- _wandbruntime‚2719  _runtime‚2719¶HÙžaØ«¥ ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.03966021537780762 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1746031790971756 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.0792718306183815 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 *"Purity/1FRt*SRqqt4FBt/event_purity‚0.5 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2708333432674408 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1979166716337204 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9879435896873474 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21522986888885498 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.00879429280757904 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.44117647409439087 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994527101516724 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.10262377560138702 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3740827739238739 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.07144434005022049 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.414743572473526 )jet/accuracy_1_of_5‚0.895622193813324 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.15625 particle/accuracy_1_of_7‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.23192158341407776 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.28955569863319397 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.17995496094226837 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9934276938438416 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1965354084968567 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 jet/accuracy_4_of_6‚0 jet/accuracy_2_of_7‚0 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_3‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.32499998807907104 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.293279230594635 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.27360135316848755 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19416643679141998 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 +particle/specificity‚0.9359370470046997 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2520114481449127 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.22499999403953552 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.45069271326065063 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 6 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.320175439119339 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 5 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.96120685338974 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1422642022371292 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6000000238418579 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4690661132335663 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5263752341270447 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.05660262703895569 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1185954213142395 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.24745520949363708 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18038862943649292 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚1 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1959775686264038 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14744581282138824 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.35044828057289124 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 *jet/accuracy_1_of_6‚0.9679487943649292 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.25664734840393066 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_5‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 _step‚296 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.23372317850589752 *jet/accuracy_1_of_2‚0.6360551118850708 - Purity/0FRt0SRqqt3FBt/FBt_purity‚0.921875 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.031598884612321854 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9919911026954651 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.006907493807375431 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3725699186325073 *loss/val_total_loss‚55.337120333757305 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.3947368562221527 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.47994154691696167 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.16749687492847443 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0014430014416575432 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7565625309944153 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06465653330087662 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.18409088253974915 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2220539003610611 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.2705152928829193 particle/accuracy_2_of_4‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996220469474792 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3947916030883789 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3166666626930237 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.2636111378669739 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.968586802482605 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.24997283518314362 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 jet/accuracy_4_of_7‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.2708333432674408 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.014172335155308247 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9809523820877075 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151560306549 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3236083686351776 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.27360135316848755 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9970759749412537 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15635301172733307 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.004704301245510578 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994527101516724 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21094843745231628 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6363636255264282 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.2198198437690735 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.29012346267700195 +jet/accuracy_2_of_6‚0.21153846383094788 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9838148951530457 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17071755230426788 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.29384303092956543 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0009920635493472219 (particle/accuracy‚0.8329972624778748 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_7‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.15350878238677979 jet/accuracy_6_of_7‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.40132036805152893 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.28617724776268005 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3700000047683716 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18893548846244812 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9902305006980896 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.12347035855054855 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18391653895378113 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.31298449635505676 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.11191906034946442 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.0781220942735672 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.24335508048534393 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.2594839632511139 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6262714862823486 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.3024691045284271 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6111111044883728 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3067419230937958 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.031931646168231964 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.2886987030506134 particle/accuracy_5_of_7‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18559974431991577 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02589285746216774 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9921502470970154 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2787037193775177 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.14707602560520172 7validation_average_jet_accuracy‚0.34898969531059265 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20157434046268463 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4305618703365326 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25230321288108826 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 jet/accuracy_6_of_6‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19202543795108795 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.2631102204322815 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4097222089767456 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt4SRqqt0FBt/FRt_purity‚0 jet/accuracy_3_of_7‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9704861044883728 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 (particle/f_score‚0.34084340929985046 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_6_of_7‚1 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.31969180703163147 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.2519982159137726 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2731659710407257 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.28672128915786743 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 particle/accuracy_2_of_2‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4084191620349884 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.29999998211860657 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 epoch‚2 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.014136364683508873 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8209401965141296 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.2681904435157776 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5166900753974915 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.16687482595443726 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3448276221752167 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.033565204590559006 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.13271604478359222 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09261643886566162 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.24406257271766663 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.08666667342185974 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 _runtime‚2719.901708686 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2658737897872925 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973127841949463 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 particle/accuracy_4_of_5‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9817396402359009 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.42666664719581604 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.2710397243499756 +jet/accuracy_3_of_5‚0.06431131064891815 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.11153645068407059 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2186199575662613 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28113532066345215 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 % _timestamp‚1.7884101094872375e+09 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4301437437534332 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1785164475440979 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9969828128814697 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.16580459475517273 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1736111044883728 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.8026440143585205 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125000298023224 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.2938530743122101 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4186507761478424 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.014213141985237598 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.30955642461776733 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.027852652594447136 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 & Purity/2FRt4SRqqt*FBt/FRt_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.49088579416275024 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.2967814803123474 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9850582480430603 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.32608503103256226 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3540724515914917 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_7‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 >&Purity/1FRt0SRqqt2FBt/ev…Ö>1ent_proportion‚0.01512896828353405 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.004715656861662865 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9832133650779724 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16933688521385193 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15040980279445648 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1103026270866394 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 particle/accuracy_6_of_6‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 jet/accuracy_7_of_7‚0 ,jet/accuracy_3_of_3‚0.017389006912708282 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6445705890655518 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.30584895610809326 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 +particle/sensitivity‚0.2766788601875305 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4976958930492401 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.326662540435791 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15397799015045166 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.014405392110347748 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_5‚0.3281489610671997 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3928571343421936 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.41235631704330444 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 particle/accuracy_5_of_6‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.24782134592533112 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3420744240283966 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 trainer/global_step‚3626 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9901803135871887 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.014720333740115166 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0035348127130419016 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1666666567325592 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4166666567325592 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5066384077072144 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18337778747081757 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.27154672145843506 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 particle/accuracy_1_of_3‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_3_of_5‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9868677854537964 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.42837977409362793 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.425653874874115 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.09253980964422226 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0007215007208287716 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2879464328289032 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3854166567325592 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.27354779839515686 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.12164750695228577 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.1424391269683838 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.26423612236976624 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4824404716491699 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1353698968887329 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.12207180261611938 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1103026270866394 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 -jet/accuracy_5_of_5‚0.0026881720405071974 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.25859788060188293 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04839257150888443 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 jet/accuracy_1_of_7‚0.5 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16638320684432983 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9974978566169739 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 jet/accuracy_5_of_6‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9887885451316833 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6363636255264282 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1726195216178894 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5350877642631531 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.36473003029823303 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.446519672870636 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.17935298383235931 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4348537027835846 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9945887327194214 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9880224466323853 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.2874695658683777 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9898810982704163 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.18858161568641663 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 *jet/accuracy_2_of_3‚0.1882537454366684 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_4‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_6‚0.03846153989434242 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.061371102929115295 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.357638955116272 particle/accuracy_1_of_6‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.18592114746570587 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9956801533699036 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.26284411549568176 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3143939673900604 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.2510456144809723 particle/accuracy_1_of_2‚1 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3690205216407776 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4375 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.2536384165287018 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_7_of_7‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1636989414691925 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.14542609453201294 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 + Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1875 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1453900784254074 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924°³åÞNÙ°¥ particle/accuracy_7_of_7‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1636989414691925 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.14542609453201294 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 + Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1875 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1453900784254074 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.03966021537780762 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1746031790971756 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.0792718306183815 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 *"Purity/1FRt*SRqqt4FBt/event_purity‚0.5 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2708333432674408 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1979166716337204 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9879435896873474 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21522986888885498 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.00879429280757904 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.44117647409439087 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994527101516724 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.10262377560138702 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3740827739238739 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.07144434005022049 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.414743572473526 )jet/accuracy_1_of_5‚0.895622193813324 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.15625 particle/accuracy_1_of_7‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.23192158341407776 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.28955569863319397 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.17995496094226837 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9934276938438416 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1965354084968567 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 jet/accuracy_4_of_6‚0 jet/accuracy_2_of_7‚0 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_3‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.32499998807907104 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.293279230594635 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.27360135316848755 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19416643679141998 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 +particle/specificity‚0.9359370470046997 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2520114481449127 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.22499999403953552 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.45069271326065063 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 6 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.320175439119339 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 5 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.96120685338974 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1422642022371292 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6000000238418579 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4690661132335663 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5263752341270447 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.05660262703895569 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1185954213142395 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.24745520949363708 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18038862943649292 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚1 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1959775686264038 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14744581282138824 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.35044828057289124 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 *jet/accuracy_1_of_6‚0.9679487943649292 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.25664734840393066 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_5‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 _step‚296 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.23372317850589752 *jet/accuracy_1_of_2‚0.6360551118850708 - Purity/0FRt0SRqqt3FBt/FBt_purity‚0.921875 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.031598884612321854 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9919911026954651 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.006907493807375431 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3725699186325073 *loss/val_total_loss‚55.337120333757305 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.3947368562221527 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.47994154691696167 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.16749687492847443 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0014430014416575432 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7565625309944153 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06465653330087662 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.18409088253974915 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2220539003610611 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.2705152928829193 particle/accuracy_2_of_4‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996220469474792 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3947916030883789 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3166666626930237 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.2636111378669739 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.968586802482605 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.24997283518314362 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 jet/accuracy_4_of_7‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.2708333432674408 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.014172335155308247 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9809523820877075 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151560306549 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3236083686351776 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.27360135316848755 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9970759749412537 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15635301172733307 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.004704301245510578 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994527101516724 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21094843745231628 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6363636255264282 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.2198198437690735 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.29012346267700195 +jet/accuracy_2_of_6‚0.21153846383094788 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9838148951530457 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17071755230426788 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.29384303092956543 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0009920635493472219 (particle/accuracy‚0.8329972624778748 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_7‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.15350878238677979 jet/accuracy_6_of_7‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.40132036805152893 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.28617724776268005 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3700000047683716 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18893548846244812 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9902305006980896 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.12347035855054855 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18391653895378113 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.31298449635505676 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.11191906034946442 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.0781220942735672 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.24335508048534393 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.2594839632511139 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6262714862823486 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.3024691045284271 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6111111044883728 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3067419230937958 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.031931646168231964 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.2886987030506134 particle/accuracy_5_of_7‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18559974431991577 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02589285746216774 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9921502470970154 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2787037193775177 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.14707602560520172 7validation_average_jet_accuracy‚0.34898969531059265 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20157434046268463 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4305618703365326 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25230321288108826 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 jet/accuracy_6_of_6‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19202543795108795 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.2631102204322815 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4097222089767456 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt4SRqqt0FBt/FRt_purity‚0 jet/accuracy_3_of_7‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9704861044883728 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 (particle/f_score‚0.34084340929985046 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_6_of_7‚1 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.31969180703163147 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.2519982159137726 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2731659710407257 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.28672128915786743 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 particle/accuracy_2_of_2‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4084191620349884 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.29999998211860657 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 epoch‚2 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.014136364683508873 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8209401965141296 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.0001860119082è‚ËÙC2664648 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.2681904435157776 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5166900753974915 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.16687482595443726 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3448276221752167 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.033565204590559006 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.13271604478359222 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09261643886566162 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.24406257271766663 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.08666667342185974 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 _runtime‚2719.901708686 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2658737897872925 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973127841949463 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 particle/accuracy_4_of_5‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9817396402359009 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.42666664719581604 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.2710397243499756 +jet/accuracy_3_of_5‚0.06431131064891815 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.11153645068407059 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2186199575662613 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28113532066345215 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 % _timestamp‚1.7884101094872375e+09 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4301437437534332 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1785164475440979 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9969828128814697 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.16580459475517273 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1736111044883728 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.8026440143585205 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125000298023224 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.2938530743122101 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4186507761478424 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.014213141985237598 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.30955642461776733 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.027852652594447136 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 & Purity/2FRt4SRqqt*FBt/FRt_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.49088579416275024 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.2967814803123474 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9850582480430603 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.32608503103256226 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3540724515914917 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_7‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.004715656861662865 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9832133650779724 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16933688521385193 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15040980279445648 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1103026270866394 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 particle/accuracy_6_of_6‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 jet/accuracy_7_of_7‚0 ,jet/accuracy_3_of_3‚0.017389006912708282 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6445705890655518 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.30584895610809326 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 +particle/sensitivity‚0.2766788601875305 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4976958930492401 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.326662540435791 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15397799015045166 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.014405392110347748 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_5‚0.3281489610671997 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3928571343421936 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.41235631704330444 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 particle/accuracy_5_of_6‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.24782134592533112 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3420744240283966 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 trainer/global_step‚3626 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9901803135871887 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.014720333740115166 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0035348127130419016 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1666666567325592 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4166666567325592 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5066384077072144 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18337778747081757 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.27154672145843506 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 particle/accuracy_1_of_3‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_3_of_5‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9868677854537964 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.42837977409362793 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.425653874874115 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.09253980964422226 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0007215007208287716 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2879464328289032 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3854166567325592 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.27354779839515686 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.12164750695228577 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.1424391269683838 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.26423612236976624 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4824404716491699 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1353698968887329 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.12207180261611938 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1103026270866394 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 -jet/accuracy_5_of_5‚0.0026881720405071974 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.25859788060188293 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04839257150888443 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 jet/accuracy_1_of_7‚0.5 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16638320684432983 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9974978566169739 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 jet/accuracy_5_of_6‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9887885451316833 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6363636255264282 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1726195216178894 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5350877642631531 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.36473003029823303 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.446519672870636 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.17935298383235931 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4348537027835846 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9945887327194214 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9880224466323853 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.2874695658683777 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9898810982704163 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.18858161568641663 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 *jet/accuracy_2_of_3‚0.1882537454366684 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_4‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_6‚0.03846153989434242 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.061371102929115295 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.357638955116272 particle/accuracy_1_of_6‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.18592114746570587 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9956801533699036 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.26284411549568176 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3143939673900604 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.2510456144809723 particle/accuracy_1_of_2‚1 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3690205216407776 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4375 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.2536384165287018 ("Purity/*FRt4SRqqt1FBt/event_purity‚0¨-¸¡Ú‚(:„ ‡òãÔ˸ԙ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.115' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.623 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý("Û‚(:— ‡òãÔœ“åœ disk.sda2.out‚0$ memory_percent‚98.45867156982422% disk./.usageGB‚1089.2373085021973 network.sent‚ 7.704603e+06 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 119.32421875 disk.sda.out‚ 119.32421875 disk.drbd1777.in‚0 cpu‚25.4066143017563 proc.cpu.threads‚17) disk./.usagePercent‚73.09962970455828& disk.nvme0n1.out‚46211.62255859375 disk.rbd2.in‚0 disk.rbd2.out‚ 64.361328125 disk.nvme0c0n1.in‚ 710.40625 disk.dm-0.in‚ 710.40625 disk.dm-1.out‚ 119.32421875 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 1.595703125 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2895.99609375) proc.memory.percent‚17.67575740814209% proc.memory.availableMB‚ 252.53125 network.recv‚ 1.016694e+06 disk.rbd1.in‚0 disk.nvme0n1.in‚ 710.40625# disk.dm-0.out‚46211.62255859375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚46211.62255859375 disk.sda2.in‚0 disk.drbd1777.out‚0nW2ë2Ü- _wandbruntime‚2738  _runtime‚2738 1îoÇÝÁ _step‚297 & lr-AdamW/pg1‚8.900891365871563e-05 & lr-AdamW/pg2‚8.900891365871563e-05 trainer/global_step‚3649 % _timestamp‚1.7884101276359708e+09 _runtime‚2738.048129797ÜKûwÌÞÆ & lr-AdamW/pg1‚8.900891365871563e-05 & lr-AdamW/pg2‚8.900891365871563e-05 trainer/global_step‚3649 % _timestamp‚1.7884101276359708e+09 _runtime‚2738.048129797 _step‚297© «¡2ß- _wandbruntime‚2738  _runtime‚2738ØrnØ'àÒO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _runtime‚2738.049043569 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48238683 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5151478436352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚109248547949568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114490069430272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚103222508418250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚109248524009472 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5241521480704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5136622285002 $ _timestamp‚1.788410127636715e+09 _step‚298 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5151309436416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚108262843044352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73864170 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48238685 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49021599 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20858843 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114489837002752 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚108262843044352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48238683 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28162756 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚98085886133248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48238685 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5151309436416 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚550 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚108262656041472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20858293 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122102853 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚98085869093888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚108262656041472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73864170 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28162754 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122122893 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5151478436352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49021047 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5241312993280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚103222326257486 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5136457163598 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257I”!´h á×O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚109248524009472 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5241521480704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5136622285002 $ _timestamp‚1.788410127636715e+09 _step‚298 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5151309436416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚108262843044352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73864170 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48238685 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49021599 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20858843 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114489837002752 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚108262843044352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48238683 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28162756 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚98085886133248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48238685 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5151309436416 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QIDeviceStatsMonitor.on_tra—^Guin_batch_start/inactive_split.small_pool.current‚550 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚108262656041472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20858293 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122102853 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚98085869093888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚108262656041472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73864170 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28162754 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122122893 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5151478436352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49021047 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5241312993280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚103222326257486 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5136457163598 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _runtime‚2738.049043569 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48238683 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5151478436352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚109248547949568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114490069430272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚103222508418250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0ª^¶V­2â- _wandbruntime‚2738  _runtime‚2738§± Mâ&ãÜM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5152702574592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122155365 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28170131 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48251342 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122155365 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122135299 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48251342 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5242734798336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚ 2738.69523812 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚108291505560576 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49033788 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28170133 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48251344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚114520171756032 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5242943272448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5152871587840 _step‚299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚103138615969792 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20863657 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚98111829315584 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚108291505560576 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚3649 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚108291318544384 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚103138633972736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚103138615969792 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73904021 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 % _timestamp‚1.7884101282824352e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5152702574592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5152871587840 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73883957 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚103249840765338 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48251344 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73904021 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚109277460897792 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5137846328142 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚103249658604366 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73883957 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5138011449754 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚103138633972736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2‘¼yjç&äáM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 % _timestamp‚1.7884101282824352e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5152702574592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5152871587840 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73883957 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚103249840765338 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48251344 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73904021 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚109277460897792 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5137846328142 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚103249658604366 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73883957 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5138011449754 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚103138633972736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5152702574592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122155365 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28170131 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48251342 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122155365 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122135299 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48251342 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5242734798336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚ 2738.69523812 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚108291505560576 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49033788 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28170133 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48251344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚114520171756032 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5242943272448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5152871587840 _step‚299 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚103138615969792 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20863657 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚98111829315584 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚108291505560576 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚3649 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚108291318544384 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚103138633972736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚103138615969792 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73904021 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944«¥w<·2å- _wandbruntime‚2738  _runtime‚2738f5µÀUæÏ 1loss/SRqqt3/detection_loss‚1.4940178520977496 /loss/FBt4/detection_loss‚3.3364377319812775 0loss/FRt2/assignment_loss‚3.7631527000003393 2loss/SRqqt4/assignment_loss‚1.5959540915030699 0loss/FBt1/assignment_loss‚0.5699184137128113 /loss/FRt3/detection_loss‚1.8813314833678305 0loss/SRqqt1/detection_loss‚2.807468702395757 /loss/FBt1/detection_loss‚2.5620359083016715 trainer/global_step‚3649 0loss/FBt2/assignment_loss‚0.2741421619255561 /loss/FRt2/detection_loss‚2.3747263133525847 /loss/FRt4/detection_loss‚2.0101173722519063 /loss/SRqqt2/detection_loss‚1.62522676671055 $ _timestamp‚1.788410128284413e+09 _step‚300 2loss/SRqqt3/assignment_loss‚2.0174778874963524 .loss/FBt2/detection_loss‚4.655554461479187 /loss/FBt3/detection_loss‚0.9389307440770904 epoch‚3 2loss/SRqqt2/assignment_loss‚1.6613930633265368 1loss/SRqqt4/detection_loss‚1.6991616312700968 %loss/total_loss‚53.36673990952704 /loss/FRt1/assignment_loss‚4.817635756272536 .loss/FRt4/assignment_loss‚4.09332948360803 1loss/SRqqt1/assignment_loss‚2.342666211227576 0loss/FBt3/assignment_loss‚0.3520134170188117 /loss/FRt1/detection_loss‚3.1887894800076118 _runtime‚ 2738.69647457 0loss/FRt3/assignment_loss‚2.7881753481924534 0loss/FBt4/assignment_loss‚0.5170829279496603Ì”rZçÔ /loss/SRqqt2/detection_loss‚1.62522676671055 $ _timestamp‚1.788410128284413e+09 _step‚300 2loss/SRqqt3/assignment_loss‚2.0174778874963524 .loss/FBt2/detection_loss‚4.655554461479187 /loss/FBt3/detection_loss‚0.9389307440770904 epoch‚3 2loss/SRqqt2/assignment_loss‚1.6613930633265368 1loss/SRqqt4/detection_loss‚1.6991616312700968 %loss/total_loss‚53.36673990952704 /loss/FRt1/assignment_loss‚4.817635756272536 .loss/FRt4/assignment_loss‚4.09332948360803 1loss/SRqqt1/assignment_loss‚2.342666211227576 0loss/FBt3/assignment_loss‚0.3520134170188117 /loss/FRt1/detection_loss‚3.1887894800076118 _runtime‚ 2738.69647457 0loss/FRt3/assignment_loss‚2.7881753481924534 0loss/FBt4/assignment_loss‚0.5170829279496603 1loss/SRqqt3/detection_loss‚1.4940178520977496 /loss/FBt4/detection_loss‚3.3364377319812775 0loss/FRt2/assignment_loss‚3.7631527000003393 2loss/SRqqt4/assignment_loss‚1.5959540915030699 0loss/FBt1/assignment_loss‚0.5699184137128113 /loss/FRt3/detection_loss‚1.8813314833678305 0loss/SRqqt1/detection_loss‚2.807468702395757 /loss/FBt1/detection_loss‚2.5620359083016715 trainer/global_step‚3649 0loss/FBt2/assignment_loss‚0.2741421619255561 /loss/FRt2/detection_loss‚2.3747263133525847 /loss/FRt4/detection_loss‚2.0101173722519063¬Š˜c‘è‚(:† –òãÔÚŒø™ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.867' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9734 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jÍù é‚(:• –òãÔÒÀƒœ% proc.memory.availableMB‚ 252.59375 cpu‚25.40460786141518 disk.sda1.out‚0 disk.rbd1.out‚ 1.595703125# disk.dm-0.out‚46343.80419921875 disk.rbd0.out‚0 network.sent‚ 7.828968e+06 disk.sda3.in‚0 disk.sda.out‚ 119.7421875 proc.cpu.threads‚17 network.recv‚ 1.023718e+06( disk.nvme0c0n1.out‚46343.80419921875 disk.sda3.out‚ 119.7421875 disk.sda.in‚0 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 710.40625 disk.rbd2.out‚ 64.501953125 disk.sda2.out‚0 disk.dm-0.in‚ 710.40625 disk.dm-1.out‚ 119.7421875 disk.rbd3.in‚0 disk.rbd3.out‚0) proc.memory.percent‚17.67575740814209 disk.nvme0c0n1.in‚ 710.40625 disk.sda1.in‚0$ memory_percent‚98.45829010009766% disk./.usageGB‚1089.2368125915527 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚46343.80419921875# proc.memory.rssMB‚ 2895.99609375) disk./.usagePercent‚73.09959642357884 disk.dm-1.in‚0 disk.rbd2.in‚0¢Ö³1ê‚(:… ¥òãÔع¨™ gpu.0.gpu‚11.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s‹žì!ë‚(:š ¥òãÔà³Î› disk.sda3.in‚0 disk.dm-0.in‚ 710.40625 disk.nvme0n1.in‚ 710.40625& disk.nvme0n1.out‚46443.05224609375 disk.sda.in‚0# proc.memory.rssMB‚ 2895.99609375 disk.sda1.out‚0# disk.dm-0.out‚46443.05224609375 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 7.93759e+06% disk./.usageGB‚1089.2370109558105 disk.sda.out‚ 120.35546875 cpu‚25.402705005985975 proc.cpu.threads‚17) disk./.usagePercent‚73.09960973597063 disk.dm-1.out‚ 120.35546875 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.in‚0 network.recv‚ 1.030326e+06 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.595703125 disk.dm-1.in‚0) proc.memory.percent‚17.67575740814209 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 120.35546875 disk.rbd2.in‚0$ memory_percent‚98.45824241638184' proc.memory.availableMB‚ 252.6015625 disk.drbd1777.out‚0 disk.rbd2.out‚ 64.501953125( disk.nvme0c0n1.out‚46443.05224609375 disk.sda2.out·DòJ‚0綶J2ì- _wandbruntime‚2773  _runtime‚2773` .ÆíÀ trainer/global_step‚3699 $ _timestamp‚1.788410162738611e+09 _runtime‚2773.150809798 _step‚301 & lr-AdamW/pg1‚8.853261060835771e-05 & lr-AdamW/pg2‚8.853261060835771e-05ÍssËîÅ & lr-AdamW/pg1‚8.853261060835771e-05 & lr-AdamW/pg2‚8.853261060835771e-05 trainer/global_step‚3699 $ _timestamp‚1.788410162738611e+09 _runtime‚2773.150809798 _step‚301­“Á/2ï- _wandbruntime‚2773  _runtime‚2773 ;G Ù'ðÓO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚116006318621184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚109695982603776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 _step‚302 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚503 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21125224 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110694195360768 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚104474814841856 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28531606 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5312351493632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5221149758976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚505 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚104589139353458 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 trainer/global_step‚3699 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 123728505 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49656828 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21125727 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚109695982603776 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74876910 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110694171420672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49657333 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 % _timestamp‚1.7884101627394767e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚99383028210688 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5206094103410 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _runtime‚2773.151970515 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48871633 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74856872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚104474814841856 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123748545 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5312147200512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48871635 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48871633 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109695795600896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204293120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217k= Þ'ñØO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 trainer/global_step‚3699 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 123728505 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49656828 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21125727 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚109695982603776 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74876910 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110694171420672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49657333 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 % _timestamp‚1.7884101627394767e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚99383028210688 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5206094103410 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _runtime‚2773.151970515 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48871633 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74856872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚104474814841856 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123748545 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5312147200512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48871635 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48871633 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109695795600896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204293120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚116006318621184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚109695982603776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 _step‚302 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚503 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21125224 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110694195360768 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚104474814841856 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28531606 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5312351493632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5221149758976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚505 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚104589139353458 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚104502102209536 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49670051 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28538981 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚303 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48884292 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 % _timestamp‚1.7884101633975563e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚104616471700546 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚104616289539574 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123781017 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚104502084206592 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚99408988432384 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚110723108308992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48884294 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28538983 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5222542910464 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚508 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚104502102209536 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚104502084206592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚110723084368896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5207483268162 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5222542910464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 trainer/global_step‚3699 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74876659 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49669541 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74876659 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123760951 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74896723 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48884292 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123781017 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5313776721920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚303 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48884292 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 % _timestamp‚1.7884101633975563e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚104616471700546 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚104616289539574 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123781017 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚104502084206592 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1821609~m…(ó72 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚99408988432384 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚110723108308992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48884294 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28538983 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5222542910464 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚508 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚104502102209536 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚104502084206592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚110723084368896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5207483268162 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5222542910464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 trainer/global_step‚3699 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74876659 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49669541 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74876659 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123760951 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74896723 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48884292 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123781017 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5313776721920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚104502102209536¯—(¶ü2õ- _wandbruntime‚2773  _runtime‚2773°¯cÕXöÒ /loss/FRt2/assignment_loss‚3.666184733148481 /loss/FRt1/detection_loss‚2.7522383893237397 /loss/FRt2/detection_loss‚2.1838401585328775 trainer/global_step‚3699 .loss/FRt3/assignment_loss‚2.98193360974149 /loss/FRt3/detection_loss‚2.1902817985633525 /loss/FRt4/detection_loss‚2.3342928706213484 epoch‚3 0loss/SRqqt1/detection_loss‚2.679054519709419 1loss/SRqqt2/detection_loss‚1.4474295228719711 /loss/FBt1/detection_loss‚3.1319879706089315 /loss/FBt2/detection_loss‚3.7334963579972587 _step‚304 2loss/SRqqt4/assignment_loss‚1.7722543371936021 1loss/FBt3/assignment_loss‚0.27571161581731934 1loss/SRqqt3/detection_loss‚1.8280019920481287 /loss/FBt3/detection_loss‚0.8891285593781283 %loss/total_loss‚53.55121997723917 2loss/SRqqt3/assignment_loss‚1.9007231687074122 0loss/FBt2/assignment_loss‚0.5561950907213031 1loss/SRqqt4/detection_loss‚1.7324346571655598 _runtime‚2773.812687448 /loss/FRt4/assignment_loss‚3.900937907917555 2loss/SRqqt2/assignment_loss‚1.7759743711830658 % _timestamp‚1.7884101634005125e+09 1loss/SRqqt1/assignment_loss‚2.701001524925232 0loss/FBt1/assignment_loss‚0.5358454398074173 /loss/FBt4/assignment_loss‚0.673566848228802 /loss/FBt4/detection_loss‚2.6991448539953966 /loss/FRt1/assignment_loss‚5.209559679031372E6ì|]÷× /loss/FRt4/assignment_loss‚3.900937907917555 2loss/SRqqt2/assignment_loss‚1.7759743711830658 % _timestamp‚1.7884101634005125e+09 1loss/SRqqt1/assignment_loss‚2.701001524925232 0loss/FBt1/assignment_loss‚0.5358454398074173 /loss/FBt4/assignment_loss‚0.673566848228802 /loss/FBt4/detection_loss‚2.6991448539953966 /loss/FRt1/assignment_loss‚5.209559679031372 /loss/FRt2/assignment_loss‚3.666184733148481 /loss/FRt1/detection_loss‚2.7522383893237397 /loss/FRt2/detection_loss‚2.1838401585328775 trainer/global_step‚3699 .loss/FRt3/assignment_loss‚2.98193360974149 /loss/FRt3/detection_loss‚2.1902817985633525 /loss/FRt4/detection_loss‚2.3342928706213484 epoch‚3 0loss/SRqqt1/detection_loss‚2.679054519709419 1loss/SRqqt2/detection_loss‚1.4474295228719711 /loss/FBt1/detection_loss‚3.1319879706089315 /loss/FBt2/detection_loss‚3.7334963579972587 _step‚304 2loss/SRqqt4/assignment_loss‚1.7722543371936021 1loss/FBt3/assignment_loss‚0.27571161581731934 1loss/SRqqt3/detection_loss‚1.8280019920481287 /loss/FBt3/detection_loss‚0.8891285593781283 %loss/total_loss‚53.55121997723917 2loss/SRqqt3/assignment_loss‚1.9007231687074122 0loss/FBt2/assignment_loss‚0.5561950907213031 1loss/SRqqt4/detection_loss‚1.7324346571655598 _runtime‚2773.812687448°ØY‘ø‚(:† ´òãÔˆÇÆ™ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.508' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7016 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥xž›ù‚(:‘ ´òãÔ—Ñ• disk.drbd1777.in‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 2895.99609375 disk.sda3.out‚ 120.8828125 disk.sda1.out‚0! disk.dm-0.out‚46542.470703125 disk.rbd0.in‚0( proc.memory.availableMB‚ 252.32421875) disk./.usagePercent‚73.09961536813637% disk./.usageGB‚1089.2370948791504 disk.sda2.out‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.595703125$ memory_percent‚98.45993518829346 network.sent‚ 7.943e+06 disk.sda.in‚0 disk.rbd1.in‚0 cpu‚25.40078596214253 disk.rbd0.out‚0 disk.rbd2.out‚64.66650390625& disk.nvme0c0n1.out‚46542.470703125 network.recv‚ 1.034326e+06 disk.rbd3.out‚0) proc.memory.percent‚17.67575740814209 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda.out‚ 120.8828125 disk.dm-1.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 710.40625$ disk.nvme0n1.out‚46542.470703125 disk.dm-0.in‚ 710.40625 disk.dm-1.out‚ 120.8828125 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 710.40625›øH”œú‚(:‘ ÃòãÔõ´òš gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.872' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.974400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XSµ û‚(:‚ ÃòãÔ‹·œ disk.drbd1777.out‚0 disk.dm-0.in‚ 710.40625! disk.dm-0.out‚46542.763671875 disk.sda1.in‚0# proc.memory.rssMB‚ 2895.99609375 disk.sda1.out‚0 network.sent‚ 8.051287e+06 disk.sda2.in‚0 disk.sda3.out‚121.5 disk.dm-1.in‚0( proc.memory.availableMB‚ 252.26953125) disk./.usagePercent‚73.09961536813637& disk.nvme0c0n1.out‚46542.763671875 disk.dm-1.out‚121.5 proc.cpu.threads‚17 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda.out‚121.5$ disk.nvme0n1.out‚46542.763671875) proc.memory.percent‚17.67575740814209 network.recv‚ 1.040138e+06 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda3.in‚0 disk.rbd1.in‚0# memory_percent‚98.4602689743042 cpu‚25.399019032433152 disk.drbd1777.in‚0 disk.rbd1.out‚ 1.595703125 disk.nvme0n1.in‚ 710.40625 disk.sda2.out‚0% disk./.usageGB‚1089.2370948791504 disk.rbd2.out‚64.66650390625 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 710.40625V/‰‘ü‚(:† ÒòãÔ¬Õ¯™ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.226' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6452 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾­dŽý‚(:‘ ÒòãÔ…ÖÖ› disk.rbd2.out‚64.81494140625 disk.sda1.in‚0$ disk./.usageGB‚1089.237045288086 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.out‚0 cpu‚25.397183031581697 disk.dm-0.in‚ 710.40625# proc.memory.rssMB‚ 2895.99609375 disk.nvme0c0n1.in‚ 710.40625 network.sent‚ 8.057515e+06 disk.nvme0n1.in‚ 710.40625$ disk.nvme0n1.out‚46642.376953125$ memory_percent‚98.45874309539795& disk.nvme0c0n1.out‚46642.376953125 disk.sda3.out‚ 121.984375 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 121.984375 disk.drbd1777.in‚0 disk.rbd1.out‚ 1.595703125! disk.dm-0.out‚46642.376953125 disk.dm-1.out‚ 121.984375 disk.sda2.in‚0( proc.memory.availableMB‚ 252.51953125) proc.memory.percent‚17.67575740814209) disk./.usagePercent‚73.09961204003844 disk.rbd0.out‚0 disk.sda3.in‚0 disk.drbd1777.out‚0 network.recv‚ 1.045056e+06 disk.rbd2.in‚0ï?C2þ- _wandbruntime‚2808  _runtime‚2808á ûÇÿÁ & lr-AdamW/pg2‚8.804838524249441e-05 trainer/global_step‚3749 % _timestamp‚1.7884101979882176e+09 _runtime‚2808.400364641 _step‚305 & lr-AdamW/pg1‚8.804838524249441e-05WÂå‚Ì€Æ & lr-AdamW/pg1‚8.804838524249441e-05 & lr-AdamW/pg2‚8.804838524249441e-05 trainer/global_step‚3749 % _timestamp‚1.7884101979882176e+09 _runtime‚2808.400364641 _step‚305±ï M52- _wandbruntime‚2808  _runtime‚2808õ©¼Û'‚ÕO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 125354157 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112139842771968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5383212689920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5275565921818 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚111129122163200 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50294793 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884101979889603e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75849574 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5290821081600 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111128935160320 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚2808.401313549 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75869612 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50294239 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚3749 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5290652081664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105838283078656 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105838283078656 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5290652081664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105838301081600 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75849574 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117522827228672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204293120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49504585 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _step‚306 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚111129122163200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117523055461888 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚100680187327488 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 125374197 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5275400800414 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112139818831872 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚100680204366848 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49504583 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105838301081600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28900454 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5290821081600 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111128935160320 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚2808.401313549 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75869612 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50294239 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚3749 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5290652081664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105838283078656 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105838283078656 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5290652081664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105838301081600 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75849574 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117522827228672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204293120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49504585 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _step‚306 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚111129122163200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117523055461888 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚100680187327488 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 N@DeviceStatsMonitor.on_tra–‚¡ðÿ in_batch_start/allocation.all.allocated‚ 125374197 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5275400800414 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112139818831872 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚100680204366848 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49504583 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105838301081600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28900454 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5290821081600 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚105983102635754 trainer/global_step‚3749 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125406669 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5292045219840 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75869361 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50307011 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49517242 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75889425 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚111157597663232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49517242 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚111157784679424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105865552443392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _runtime‚2809.067874114 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125406669 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚100706130509824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21399180 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚112168755720192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125386603 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5292214233088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5292045219840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50307555 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21399722 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5276789964958 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5384661646336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112168731780096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚105865552443392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28907833 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚105982920474782 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5384448977920 % _timestamp‚1.7884101986450088e+09 _step‚307¸~ßÌê&†äM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49517244 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚105983102635754 trainer/global_step‚3749 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125406669 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5292045219840 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75869361 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50307011 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49517242 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75889425 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚111157597663232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49517242 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚111157784679424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105865552443392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _runtime‚2809.067874114 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125406669 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚100706130509824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21399180 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚112168755720192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125386603 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5292214233088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5292045219840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50307555 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21399722 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5276789964958 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5384661646336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112168731780096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚105865552443392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28907833 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚105982920474782 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5384448977920 % _timestamp‚1.7884101986450088e+09 _step‚307 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105865570446336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚542 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117553180758016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5292214233088 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚100706147549184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5276955086570 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75889425 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75869361³×é²2‡- _wandbruntime‚2809  _runtime‚2809±ì'ûUˆÏ 0loss/FBt3/assignment_loss‚0.2630631643583385 0loss/SRqqt3/detection_loss‚1.840589980002154 .loss/FBt4/detection_loss‚2.657538725779607 0loss/SRqqt4/assignment_loss‚1.88512566590851 /loss/FRt1/detection_loss‚2.7982499857206604 .loss/FBt2/detection_loss‚4.084018369515737 epoch‚3 _runtime‚2809.069145259 /loss/FRt2/detection_loss‚2.1800307978283273 1loss/SRqqt1/detection_loss‚2.6196633269046914 2loss/SRqqt3/assignment_loss‚1.8588647287000308 0loss/FBt4/assignment_loss‚0.4233628039577385 0loss/SRqqt4/detection_loss‚1.611216215653853 0loss/FRt2/assignment_loss‚3.3078365434299815 0loss/FRt3/assignment_loss‚3.0286252066493033 0loss/FBt1/assignment_loss‚0.4045367725080081 %loss/total_loss‚51.60068538038814 1loss/SRqqt2/detection_loss‚1.5752352690928941 .loss/FBt1/detection_loss‚2.580906752277823 /loss/FBt3/detection_loss‚0.7802006375845141 _step‚308 /loss/FRt1/assignment_loss‚4.614486365704923 1loss/SRqqt1/assignment_loss‚2.522064219261038 2loss/SRqqt2/assignment_loss‚1.5088189508427272 1loss/FBt2/assignment_loss‚0.42278847843408585 /loss/FRt4/detection_loss‚2.0510372069417215 /loss/FRt4/assignment_loss‚4.248006881499777 /loss/FRt3/detection_loss‚2.3344183318316936 trainer/global_step‚3749 $ _timestamp‚1.788410198657054e+09±Øt)Z‰Ô 2loss/SRqqt3/assignment_loss‚1.8588647287000308 0loss/FBt4/assignment_loss‚0.4233628039577385 0loss/SRqqt4/detection_loss‚1.611216215653853 0loss/FRt2/assignment_loss‚3.3078365434299815 0loss/FRt3/assignment_loss‚3.0286252066493033 0loss/FBt1/assignment_loss‚0.4045367725080081 %loss/total_loss‚51.60068538038814 1loss/SRqqt2/detection_loss‚1.5752352690928941 .loss/FBt1/detection_loss‚2.580906752277823 /loss/FBt3/detection_loss‚0.7802006375845141 _step‚308 /loss/FRt1/assignment_loss‚4.614486365704923 1loss/SRqqt1/assignment_loss‚2.522064219261038 2loss/SRqqt2/assignment_loss‚1.5088189508427272 1loss/FBt2/assignment_loss‚0.42278847843408585 /loss/FRt4/detection_loss‚2.0510372069417215 /loss/FRt4/assignment_loss‚4.248006881499777 /loss/FRt3/detection_loss‚2.3344183318316936 trainer/global_step‚3749 $ _timestamp‚1.788410198657054e+09 0loss/FBt3/assignment_loss‚0.2630631643583385 0loss/SRqqt3/detection_loss‚1.840589980002154 .loss/FBt4/detection_loss‚2.657538725779607 0loss/SRqqt4/assignment_loss‚1.88512566590851 /loss/FRt1/detection_loss‚2.7982499857206604 .loss/FBt2/detection_loss‚4.084018369515737 epoch‚3 _runtime‚2809.069145259 /loss/FRt2/detection_loss‚2.1800307978283273 1loss/SRqqt1/detection_loss‚2.6196633269046914´ŽJcœŠ‚(:‘ áòãÔ“§¦™ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚109.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.917599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0° ã‚‹‚(:” áòãÔ¼³Åœ disk.sda1.out‚0 disk.drbd1777.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 122.6171875% disk./.usageGB‚1089.2370491027832 disk.drbd1777.in‚0 disk.rbd1.out‚ 1.595703125 disk.rbd0.in‚0& disk.nvme0c0n1.out‚46741.451171875# proc.memory.rssMB‚ 2798.62109375 network.sent‚ 8.063573e+06 disk.sda1.in‚0 disk.dm-1.in‚0) disk./.usagePercent‚73.09961229604596 disk.sda.in‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚46741.451171875 disk.dm-1.out‚ 122.6171875 disk.rbd3.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 349.734375 network.recv‚ 1.049407e+06 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 710.40625 cpu‚25.395248579972186 proc.cpu.threads‚17 disk.dm-0.in‚ 710.40625* proc.memory.percent‚17.081427574157715 disk.sda.out‚ 122.6171875 disk.nvme0n1.in‚ 710.40625! disk.dm-0.out‚46741.451171875 disk.sda2.in‚0 disk.rbd2.out‚66.32666015625$ memory_percent‚97.86539077758789¼¤€ŸŒ‚(:… ðòãÔ’†´™ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.792' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7584 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Is(u‚(:‹ ðòãÔÔ§œ disk.rbd2.out‚66.40478515625 disk.sda2.out‚0$ memory_percent‚97.86550998687744 proc.memory.rssMB‚2798.625' disk.nvme0c0n1.out‚46841.2412109375* proc.memory.percent‚17.081451416015625 disk.rbd3.in‚0% disk./.usageGB‚1089.2373161315918 disk.sda.out‚123.0625 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 710.40625( proc.memory.availableMB‚ 349.71484375 disk.sda1.out‚0 disk.dm-0.in‚ 710.40625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 710.40625 proc.cpu.threads‚17 disk.sda3.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚46841.2412109375 disk.rbd0.in‚0 network.recv‚ 1.056472e+06) disk./.usagePercent‚73.09963021657336 disk.sda3.out‚123.0625 disk.sda.in‚0" disk.dm-0.out‚46841.2412109375 disk.sda2.in‚0 network.sent‚ 8.171881e+06 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.rbd1.out‚ 1.595703125 disk.rbd3.out‚0 cpu‚25.393354412318587 disk.dm-1.in‚0 disk.dm-1.out‚123.0625E#õ]2Ž- _wandbruntime‚2843  _runtime‚2843@ëÅéÇÁ & lr-AdamW/pg1‚8.755633847296614e-05 & lr-AdamW/pg2‚8.755633847296614e-05 trainer/global_step‚3799 % _timestamp‚1.7884102331307905e+09 _runtime‚2843.542902497 _step‚309|ÌäèÌÆ & lr-AdamW/pg1‚8.755633847296614e-05 & lr-AdamW/pg2‚8.755633847296614e-05 trainer/global_step‚3799 % _timestamp‚1.7884102331307905e+09 _runtime‚2843.542902497 _step‚309µÝSÇ~2‘- _wandbruntime‚2843  _runtime‚2843m2W ’ÔO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚112562261722624 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50935868 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29269306 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚112562074719744 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5344872618822 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50137535 SDDeviceStatsMÄ3ñ½ƒonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 % _timestamp‚1.7884102331315017e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚107201751315456 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚107201769318400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5454110287360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29269304 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50935311 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 trainer/global_step‚3799 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5360323404288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76862314 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚557 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚107322219063110 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5345037740226 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 _step‚310 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚112562074719744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚555 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76862314 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5453897605632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚107201769318400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚107322401223874 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76842276 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚119039600470528 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚113585466243072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5360323404288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21666007 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚107322401223874 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76842276 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚119039600470528 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚113585466243072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5360323404288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21666007 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚112562261722624 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50935868 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29269306 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚112562074719744 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5344872618822 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50137535 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 % _timestamp‚1.7884102331315017e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚107201751315456 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚107201769318400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5454110287360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29269304 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50935311 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 trainer/global_step‚3799 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5360323404288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76862314 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚557 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚107322219063110 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5345037740226 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 _step‚310 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚112562074719744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚555 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76862314 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5453897605632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚107201769318400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 % _timestamp‚1.7884102337860498e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50150192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚107229020680192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚119069931884544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚107349733570962 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76862063 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5361885555712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5346261783366 trainer/global_step‚3799 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50948103 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29276681 _step‚311 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5361885555712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚107229038683136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127012255 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127012255 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50150194 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29276683 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21671422 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50150194 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚102003289626624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5455313987584 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚102003306665984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚107349551409990 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚107229038683136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76882127 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚112590924238848 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 2844.19876802 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚112590737222656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5361716542464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76862063 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127032321 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50948634 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119069693178880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5361716542464 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248dYø1e–ãM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76862063 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5361885555712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5346261783366 trainer/global_step‚3799 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50948103 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29276681 _step‚311 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5361885555712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚107229038683136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127012255 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127012255 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50150194 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29276683 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21671422 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50150194 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚102003289626624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5455313987584 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚102003306665984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚107349551409990 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚107229038683136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76882127 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚112590924238848 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 2844.19876802 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚112590737222656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 A9DeviceStatsMonitor.ooC„n_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5361716542464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76862063 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127032321 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50948634 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119069693178880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5361716542464 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5455528753152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76882127 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚113614403131392 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 % _timestamp‚1.7884102337860498e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50150192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚107229020680192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚119069931884544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚107349733570962·ŽÕ”2—- _wandbruntime‚2844  _runtime‚28440§ÌZ˜Ô 1loss/SRqqt4/detection_loss‚1.9931336572127683 _step‚312 0loss/SRqqt1/assignment_loss‚2.08155294598052 1loss/SRqqt3/detection_loss‚1.3647215347398411 /loss/FBt4/detection_loss‚2.5600486675898235 /loss/FRt1/assignment_loss‚4.280696332454681 /loss/FRt4/assignment_loss‚4.311889710633651 2loss/SRqqt2/assignment_loss‚1.8342011593696144 2loss/SRqqt3/assignment_loss‚1.9765692981806668 0loss/FBt2/assignment_loss‚0.6052101798343756 /loss/FBt3/detection_loss‚0.6675933023672785 % _timestamp‚1.7884102337878563e+09 _runtime‚2844.199901186 .loss/FRt2/detection_loss‚2.672996984078334 1loss/SRqqt1/detection_loss‚2.6740882432822026 1loss/FBt3/assignment_loss‚0.26571783579768155 0loss/FBt4/assignment_loss‚0.5263224362447848 /loss/FRt4/detection_loss‚2.2708562236765157 1loss/SRqqt2/detection_loss‚1.4542897439872224 /loss/FBt1/detection_loss‚2.9766373872756957 .loss/FBt2/detection_loss‚3.950963628292084 %loss/total_loss‚51.35238379602487 trainer/global_step‚3799 0loss/FRt3/assignment_loss‚1.9871124525864918 /loss/FRt3/detection_loss‚1.8458818372871195 epoch‚3 0loss/FRt2/assignment_loss‚3.3674376973739037 2loss/SRqqt4/assignment_loss‚1.9962021027292525 0loss/FBt1/assignment_loss‚0.4113959499515962 /loss/FRt1/detection_loss‚3.2768644850987654çpƧ_™Ù trainer/global_step‚3799 0loss/FRt3/assignment_loss‚1.9871124525864918 /loss/FRt3/detection_loss‚1.8458818372871195 epoch‚3 0loss/FRt2/assignment_loss‚3.3674376973739037 2loss/SRqqt4/assignment_loss‚1.9962021027292525 0loss/FBt1/assignment_loss‚0.4113959499515962 /loss/FRt1/detection_loss‚3.2768644850987654 1loss/SRqqt4/detection_loss‚1.9931336572127683 _step‚312 0loss/SRqqt1/assignment_loss‚2.08155294598052 1loss/SRqqt3/detection_loss‚1.3647215347398411 /loss/FBt4/detection_loss‚2.5600486675898235 /loss/FRt1/assignment_loss‚4.280696332454681 /loss/FRt4/assignment_loss‚4.311889710633651 2loss/SRqqt2/assignment_loss‚1.8342011593696144 2loss/SRqqt3/assignment_loss‚1.9765692981806668 0loss/FBt2/assignment_loss‚0.6052101798343756 /loss/FBt3/detection_loss‚0.6675933023672785 % _timestamp‚1.7884102337878563e+09 _runtime‚2844.199901186 .loss/FRt2/detection_loss‚2.672996984078334 1loss/SRqqt1/detection_loss‚2.6740882432822026 1loss/FBt3/assignment_loss‚0.26571783579768155 0loss/FBt4/assignment_loss‚0.5263224362447848 /loss/FRt4/detection_loss‚2.2708562236765157 1loss/SRqqt2/detection_loss‚1.4542897439872224 /loss/FBt1/detection_loss‚2.9766373872756957 .loss/FBt2/detection_loss‚3.950963628292084 %loss/total_loss‚51.35238379602487¸8\Œ‘š‚(:† ÿòãÔÍø·™ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.543' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1086 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½Ý›‚(:“ ÿòãÔóˆûœ$ disk./.usageGB‚1089.237377166748 disk.rbd0.out‚0 disk.drbd1777.in‚0 cpu‚25.391637751216017 network.recv‚ 1.06006e+06' disk.nvme0c0n1.out‚46841.7021484375 disk.nvme0n1.in‚ 710.40625 disk.dm-0.in‚ 710.40625 disk.dm-1.out‚ 123.79296875 disk.sda3.in‚0) disk./.usagePercent‚73.09963431269391 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚66.40478515625 disk.sda2.out‚0" disk.dm-0.out‚46841.7021484375" memory_percent‚97.865891456604( proc.memory.availableMB‚ 349.65234375 network.sent‚ 8.177371e+06 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.drbd1777.out‚0* proc.memory.percent‚17.081451416015625 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda3.out‚ 123.79296875 disk.rbd1.out‚ 1.595703125% disk.nvme0n1.out‚46841.7021484375 proc.memory.rssMB‚2798.625 disk.sda.in‚0 disk.sda.out‚ 123.79296875 disk.sda1.in‚0 disk.sda1.out‚0IË‘”›œ‚(: ŽóãÔäÅ™ gpu.0.gpu‚27.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚89.683' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.936600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„m,µ‚(:‡ ŽóãԬس* proc.memory.percent‚17.081451416015625% disk./.usageGB‚1089.2374000549316 disk.sda.in‚0& disk.nvme0n1.out‚46941.11865234375 disk.dm-0.in‚ 710.40625 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda.out‚124.125 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd2.out‚ 66.5849609375$ memory_percent‚97.86739349365234 network.sent‚ 8.286297e+06 disk.drbd1777.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚46941.11865234375% proc.memory.availableMB‚ 349.40625 proc.memory.rssMB‚2798.625 disk.rbd1.out‚ 1.595703125 disk.nvme0n1.in‚ 710.40625# disk.dm-0.out‚46941.11865234375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 cpu‚25.389769812070874 network.recv‚ 1.067548e+06 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda3.out‚124.125) disk./.usagePercent‚73.09963584873913 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚124.125 disk.sda2.in‚0èlÙ02ž- _wandbruntime‚2878  _runtime‚2878³±æÇŸÁ trainer/global_step‚3849 % _timestamp‚1.7884102683927846e+09 _runtime‚2878.804966817 _step‚313 & lr-AdamW/pg1‚8.705657284158222e-05 & lr-AdamW/pg2‚8.705657284158222e-05H…¿×Ì Æ & lr-AdamW/pg2‚8.705657284158222e-05 trainer/global_step‚3849 % _timestamp‚1.7884102683927846e+09 _runtime‚2878.804966817 _step‚313 & lr-AdamW/pg1‚8.705657284158222e-05¹Ë¢åù2¡- _wandbruntime‚2878  _runtime‚2878ä'›ÓÚ'¢ÔO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128605461 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50770485 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21935263 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚2878.805885809 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77834978 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚498 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚120555881525760 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50770483 _step‚314 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113995214279168 trainer/global_step‚3849 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5429994726912 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51573417 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50770483 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5429994726912 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚108565219552256 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77834978 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21935759 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚120556116050432 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113995401282048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚108565237555200 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚115031113654272 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50770485 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚108689032159082 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5430163726848 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚108565237555200 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5414509558634 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5414344437230 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77855016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77855016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128625501 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29638154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚103274505561088 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚113995214279168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚103274522600448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚108688849998318 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51573915 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969|"‘>ß'£ÙO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50770485 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚108689032159082 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5430163726848 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚108565237555200 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5414509558634 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5414344437230 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77855016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77855016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128625501 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29638154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚103274505561088 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚113995214279168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚103274522600448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚108688849998318 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51573915 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128605461 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50770485 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21935263 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚2878.805885809 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77834978 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚498 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚120555881525760 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50770483 _step‚314 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113995214279168 trainer/global_step‚3849 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5429994726912 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51573417 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50770483 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5429994726912 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚108565219552256 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77834978 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21935759 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚120556116050432 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113995401282048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚108565237555200 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚115031113654272º‡0.42¤- _wandbruntime‚2879  _runtime‚2879‡B¯.ã¥ßM P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5415733601774 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50783142 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77874829 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚103300465782784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51586654 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚115060026602496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128637907 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚494 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚120586446495744 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚108716364506170 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 % _timestamp‚1.7884102690708628e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50783144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128657973 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77874829 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29645531 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51586158 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚114024063798272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚108716182345198 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _step‚315 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128657973 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77854765 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21941121 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50783142 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚114023876782080 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5431556878336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚114024063798272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5526187479040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚2879.483578063 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128637907 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50783144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5415898723386 trainer/global_step‚3849 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚120586214081536 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚108592506919936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29645533 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401½¡Óê&¦äM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚108592488916992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5431387865088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5415733601774 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50783142 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77874829 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚103300465782784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51586654 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚115060026602496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128637907 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚494 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚120586446495744 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚108716364506170 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 % _timestamp‚1.7884102690708628e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50783144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128657973 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77874829 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29645531 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51586158 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚114024063798272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚108716182345198 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _step‚315 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128657973 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77854765 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21941121 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50783142 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚114023876782080 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5431556878336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚114024063798272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5526187479040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚2879.483578063 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128637907 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50783144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5415898723386 trainer/global_step‚3849 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚120586214081536 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚108592506919936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29645533 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115428353838592 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚109928705792000 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚109928687789056 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78847718 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51403433 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30007004 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122072679987712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚115428353838592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5499666049536 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22204978 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104571664677888 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚110055480933526 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130251153 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚318 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116476761065472 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5483816255638 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30007006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218973184 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5595676008960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5499835049472 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52211982 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104571681717248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78827680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 _runtime‚2914.204668927 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚109928705792000 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22205498rÚÊ=ß'µÙO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30007006 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218973184 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5595676008960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5499835049472 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52211982 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104571681717248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78827680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 _runtime‚2914.204668927 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚109928705792000 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22205498 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51403433 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78847718 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚3899 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116476785005568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115428540841472 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51403435 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚109928687789056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52212504 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115428353838592 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚109928705792000 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚109928687789056 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78847718 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51403433 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30007004 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122072679987712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚115428353838592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5499666049536 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22204978 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104571664677888 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚110055480933526 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130251153 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚318 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116476761065472 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5483816255638¾:b=2¶- _wandbruntime‚2914  _runtime‚2914+ºkÿä&·ÞM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78847467 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78867531 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22210545 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5597085668352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 trainer/global_step‚3899 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5501059187712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78867531 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚115457203357696 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5501228200960 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚109955957153792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130263559 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130283625 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5501059187712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚110082813280406 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51416092 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130283625 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5501228200960 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109955975156736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22211070 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 221057024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚115457016341504 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244997120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116505674013696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30014383 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51416094 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 _step‚319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30014381 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115457203357696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116505697953792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5597306725376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51416092 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚110082995441378 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122103004679168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚122102759682048 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109955975156736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78847467 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52225453 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52224926 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 $ _timestamp‚1.788410304473676e+09 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5485370541794 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130263559 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408Ýt0õÆ!¸ãM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5485370541794 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130263559 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78847467 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78867531 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22210545 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5597085668352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 trainer/global_step‚3899 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5501059187712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78867531 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚115457203357696 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5501228200960 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚109955957153792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130263559 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130283625 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5501059187712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚110082813280406 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51416092 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130283625 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5501228200960 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109955975156736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22211070 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 221057024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚115457016341504 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244997120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116505674013696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30014383 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51416094 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 _step‚319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30014381 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115457203357696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116505697953792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5597306725376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51416092 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚110082995441378 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122103004679168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚122102759682048 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109955975156736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78847467 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52225453 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52224926 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 $ _timestamp‚1.788410304473676e+09¿ßΰ¬2¹- _wandbruntime‚2914  _runtime‚29143%ËŸWºÑ 0loss/FRt3/assignment_loss‚2.5344967577192516 .loss/FRt2/detection_loss‚2.340965057412783 /loss/FRt3/detection_loss‚2.0886291144070803 1loss/SRqqt2/assignment_loss‚1.593785383179784 2loss/SRqqt3/assignment_loss‚1.9928857964721132 1loss/SRqqt2/detection_loss‚1.4587006798014044 /loss/FBt1/detection_loss‚3.1976156102286444 /loss/FRt4/assignment_loss‚3.998917872707049 .loss/FBt2/detection_loss‚5.540801763534546 .loss/FRt1/assignment_loss‚5.28549954397925 2loss/SRqqt1/assignment_loss‚2.5506814878720503 0loss/FBt2/assignment_loss‚1.6739636659622192 /loss/FRt1/detection_loss‚2.9241202884706956 /loss/FBt4/detection_loss‚2.6693478605963965 &loss/total_loss‚55.736209479860506 _runtime‚ 2914.88750744 0loss/FRt2/assignment_loss‚3.2650480791926384 1loss/SRqqt1/detection_loss‚2.9784687069746165 1loss/SRqqt3/detection_loss‚1.6010458022356033 epoch‚3 % _timestamp‚1.7884103044754653e+09 _step‚320 0loss/FBt1/assignment_loss‚0.5494333018044321 .loss/FRt4/detection_loss‚2.419410308202108 /loss/FBt3/detection_loss‚0.7433027763230106 trainer/global_step‚3899 2loss/SRqqt4/assignment_loss‚1.6772612044587731 0loss/FBt4/assignment_loss‚0.6055076354148862 1loss/FBt3/assignment_loss‚0.31784247786937997 /loss/SRqqt4/detection_loss‚1.72847830504179H|F4\»Ö 1loss/SRqqt3/detection_loss‚1.6010458022356033 epoch‚3 % _timestamp‚1.7884103044754653e+09 _step‚320 0loss/FBt1/assignment_loss‚0.5494333018044321 .loss/FRt4/detection_loss‚2.419410308202108 /loss/FBt3/detection_loss‚0.7433027763230106 trainer/global_step‚3899 2loss/SRqqt4/assignment_loss‚1.6772612044587731 0loss/FBt4/assignment_loss‚0.6055076354148862 1loss/FBt3/assignment_loss‚0.31784247786937997 /loss/SRqqt4/detection_loss‚1.72847830504179 0loss/FRt3/assignment_loss‚2.5344967577192516 .loss/FRt2/detection_loss‚2.340965057412783 /loss/FRt3/detection_loss‚2.0886291144070803 1loss/SRqqt2/assignment_loss‚1.593785383179784 2loss/SRqqt3/assignment_loss‚1.9928857964721132 1loss/SRqqt2/detection_loss‚1.4587006798014044 /loss/FBt1/detection_loss‚3.1976156102286444 /loss/FRt4/assignment_loss‚3.998917872707049 .loss/FBt2/detection_loss‚5.540801763534546 .loss/FRt1/assignment_loss‚5.28549954397925 2loss/SRqqt1/assignment_loss‚2.5506814878720503 0loss/FBt2/assignment_loss‚1.6739636659622192 /loss/FRt1/detection_loss‚2.9241202884706956 /loss/FBt4/detection_loss‚2.6693478605963965 &loss/total_loss‚55.736209479860506 _runtime‚ 2914.88750744 0loss/FRt2/assignment_loss‚3.2650480791926384 1loss/SRqqt1/detection_loss‚2.9784687069746165À&ÿû›¼‚(: ÊóãÔ¶ô¯™ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.937' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.787399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Èh(:Š ÊóãÔ‹…Çœ disk.drbd1777.out‚0 disk.rbd1.in‚0) proc.memory.percent‚17.08226203918457" disk.dm-0.out‚48198.6220703125 disk.dm-1.out‚126.4375 disk.nvme0c0n1.in‚ 710.40625 disk.sda.out‚126.4375 proc.cpu.threads‚17 disk.dm-0.in‚ 710.40625 disk.dm-1.in‚0 disk.rbd2.in‚0# memory_percent‚98.6208438873291 disk.nvme0n1.in‚ 710.40625 disk.rbd3.out‚0 disk.sda2.in‚0 disk.drbd1777.in‚0 cpu‚25.382873226998804) disk./.usagePercent‚73.16278778728551% disk.nvme0n1.out‚48198.6220703125 disk.sda.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 225.9609375$ disk./.usageGB‚1090.178409576416 disk.rbd0.in‚0' disk.nvme0c0n1.out‚48198.6220703125 disk.sda2.out‚0" proc.memory.rssMB‚ 2798.7578125 network.recv‚ 1.087435e+06 disk.rbd3.in‚0 disk.sda3.out‚126.4375 disk.rbd1.out‚ 1.595703125 network.sent‚ 8.412362e+06 disk.rbd2.out‚ 68.3779296875 disk.sda1.in‚0”û~›¾‚(: ÙóãÔ™­™ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚89.302' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.860400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡.¿‚(:“ ÙóãÔ´±Ñœ disk.rbd3.out‚0 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 126.953125 disk.rbd2.in‚0) disk./.usagePercent‚73.16278650724783 disk.sda3.out‚ 126.953125 disk.drbd1777.out‚0 disk.rbd0.out‚0$ memory_percent‚98.62077236175537% disk./.usageGB‚1090.1783905029297 disk.sda.in‚0 disk.nvme0n1.in‚ 710.40625% disk.nvme0n1.out‚49115.7900390625) proc.memory.percent‚17.08226203918457 disk.rbd1.out‚ 1.595703125' disk.nvme0c0n1.out‚49115.7900390625" proc.memory.rssMB‚ 2798.7578125 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd2.out‚ 68.3779296875 disk.rbd3.in‚0( proc.memory.availableMB‚ 225.97265625 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 710.40625 cpu‚25.381187831889175 network.sent‚ 8.520473e+06 network.recv‚ 1.094118e+06 disk.sda2.out‚0 disk.sda.out‚ 126.953125 disk.dm-0.in‚ 710.40625" disk.dm-0.out‚49115.7900390625SœfÇ2À- _wandbruntime‚2949  _runtime‚2949p#¾€ÅÁ¿ _runtime‚2949.877911924 _step‚321 % lr-AdamW/pg1‚8.60343031817759e-05 % lr-AdamW/pg2‚8.60343031817759e-05 trainer/global_step‚3949 % _timestamp‚1.7884103394657354e+09ÍØ©¦ÊÂÄ _step‚321 % lr-AdamW/pg1‚8.60343031817759e-05 % lr-AdamW/pg2‚8.60343031817759e-05 trainer/global_step‚3949 % _timestamp‚1.7884103394657354e+09 _runtime‚2949.877911924Á'ëÿ2Ã- _wandbruntime‚2949  _runtime‚2949M †Û'ÄÕO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚111292174028800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52851033 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5569337372160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52036385 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚111292156025856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5666767952384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 % _timestamp‚1.7884103394665084e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5569337372160 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131876805 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22475715 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚116861680400896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30375856 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚117922432416768 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5553453195450 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22475179 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚123588965844480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _step‚322 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚111292156025856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111292174028800 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚123589200369152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131856765 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79840420 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚105868823794688 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52036383 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52036383 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚3949 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚116861680400896 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79820382 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5553288074046 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52036385 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111422111868734 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚117922408476672 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52851571 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111422294029498 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚116861493398016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5569506372096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5666557367808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0¢n7(à'ÅÚO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚111292174028800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52851033 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5569337372160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52036385 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚111292156025856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5666767952384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 % _timestamp‚1.7884103394665084e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5569337372160 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131876805 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22475715 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚116861680400896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30375856 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚117922432416768 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5553453195450 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22475179 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚123588965844480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _step‚322 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚111292156025856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111292174028800 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚123589200369152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131856765 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79840420 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚105868823794688 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52036383 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52036383 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚3949 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚116861680400896 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79820382 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5553288074046 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52036385 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111422111868734 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚117922408476672 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52851571 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111422294029498 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚116861493398016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5569506372096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5666557367808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0Â4•õU2Æ- _wandbruntime‚2950  _runtime‚2950;$¶–?ÇßM FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30383231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚117951321424896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52864191 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚105894766977024 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131889211 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚123619296798720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131909277 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5554677238590 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _runtime‚2950.535765894 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3949 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131909277 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22480457 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5554842360202 % _timestamp‚1.7884103401231482e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚116890342917120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚503 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111449444215614 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5570730510336 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 YGDeviceStatsMonitor.onxDg¦_train_batch_end/active_bytes.small_pool.allocated‚ 5570899523584 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131889211 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79860233 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52049044 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111449626376586 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5570730510336 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚123619531310080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52049042 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79840169 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30383233 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚105894784016384 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚111319425390592 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22480958 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚111319443393536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚501 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52863688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5667975373824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚116890155900928 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47572452575¬Úêê&ÈäM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52863688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5667975373824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚116890155900928 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30383231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚117951321424896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52864191 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚105894766977024 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131889211 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚123619296798720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131909277 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5554677238590 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _runtime‚2950.535765894 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3949 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131909277 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22480457 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5554842360202 % _timestamp‚1.7884103401231482e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚116890342917120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚503 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111449444215614 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5570730510336 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5570899523584 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131889211 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79860233 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52049044 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111449626376586 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5570730510336 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚123619531310080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52049042 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79840169 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30383233 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚105894784016384 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚111319425390592 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22480958 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚111319443393536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚501 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53489103 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884103747118332e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚107165999950848 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22744399 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80813084 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5639008694784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5737455193088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80833122 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚107165982911488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80813084 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚118294632957440 _runtime‚2985.124183398 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚118294632957440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚119368055887872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚112655642265600 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚112655624262656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22744939 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚118294819960320 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230330368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040&Rð†Ï E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133482417 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚119368079827968®~O]à'×ÚO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚107165982911488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80813084 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚118294632957440 _runtime‚2985.124183398 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚118294632957440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚119368055887872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚112655642265600 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚112655624262656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22744939 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚118294819960320 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230330368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133482417 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚119368079827968 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53489103 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884103747118332e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚107165999950848 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22744399 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80813084 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5639008694784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5737455193088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80833122 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160Æ1¿2Ø- _wandbruntime‚2985  _runtime‚2985kRã&ÙÝM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚112682893627392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5640401832960 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚118323482476544 trainer/global_step‚3999 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112682893627392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22750075 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5624314178610 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _step‚327 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚112682911630336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52681994 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133514863 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22750611 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5640570846208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53502156 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52681994 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80852935 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5624149056998 _runtime‚ 2985.78487352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133534929 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52681992 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80832871 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚107191943133184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚125135835985408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5739081914880 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53502694 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚112816257311794 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚112816075150822 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30752083 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133514863 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80832871 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚119396992776192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚119396968836096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133534929 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30752081 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5738867149312 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚107191926093824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5640570846208 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80852935 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52681992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚118323482476544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199G‰‚è&ÚâM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚112682893627392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5640401832960 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚118323482476544 trainer/global_step‚3999 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112682893627392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22750075 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5624314178610 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _step‚327 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚112682911630336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52681994 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133514863 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22750611 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5640570846208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53502156 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52681994 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80852935 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5624149056998 _runtime‚ 2985.78487352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133534929 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52681992 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80832871 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚107191943133184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚125135835985408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5739081914880 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53502694 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚112816257311794 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚112816075150822 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30752083 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133514863 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80832871 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚119396992776192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚119396968836096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133534929 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30752081 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5738867149312 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚107191926093824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5640570846208 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80852935 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52681992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚118323482476544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ÇEi™õ2Û- _wandbruntime‚2985  _runtime‚2985ûA¨|UÜÏ /loss/FBt3/detection_loss‚0.7513084221401567 -loss/FRt4/detection_loss‚2.16330669231193 /loss/FBt1/detection_loss‚3.1034642566334116 /loss/FBt2/detection_loss‚3.3620307530675615 epoch‚3 /loss/FRt4/assignment_loss‚4.437832966793415 _step‚328 0loss/FRt3/assignment_loss‚2.4521083384752274 0loss/FBt3/assignment_loss‚0.3056022009454456 .loss/FRt3/detection_loss‚2.348321903158318 /loss/FBt4/detection_loss‚2.5058144964277744 /loss/FRt1/detection_loss‚3.1336853007475534 1loss/SRqqt1/detection_loss‚2.8464967577621856 0loss/FBt1/assignment_loss‚0.4303043636530426 1loss/FBt2/assignment_loss‚0.40991163573094774 %loss/total_loss‚51.58395646652357 trainer/global_step‚3999 /loss/FRt1/assignment_loss‚4.589960185686747 -loss/FRt2/detection_loss‚2.12176880430668 1loss/SRqqt2/detection_loss‚1.4324813328865098 $ _timestamp‚1.788410375374166e+09 _runtime‚2985.786217733 1loss/SRqqt1/assignment_loss‚2.627455691839087 /loss/FRt2/assignment_loss‚2.785663751845664 1loss/SRqqt2/assignment_loss‚1.700154625606246 2loss/SRqqt3/assignment_loss‚1.8486149351422985 2loss/SRqqt4/assignment_loss‚1.8884610907976018 1loss/FBt4/assignment_loss‚0.46028379701101585 0loss/SRqqt3/detection_loss‚1.801029102333511 1loss/SRqqt4/detection_loss‚2.0778950612212337U›:•ÝÔ /loss/FRt4/assignment_loss‚4.437832966793415 _step‚328 0loss/FRt3/assignment_loss‚2.4521083384752274 0loss/FBt3/assignment_loss‚0.3056022009454456 .loss/FRt3/detection_loss‚2.348321903158318 /loss/FBt4/detection_loss‚2.5058144964277744 /loss/FRt1/detection_loss‚3.1336853007475534 1loss/SRqqt1/detection_loss‚2.8464967577621856 0loss/FBt1/assignment_loss‚0.4303043636530426 1loss/FBt2/assignment_loss‚0.40991163573094774 %loss/total_loss‚51.58395646652357 trainer/global_step‚3999 /loss/FRt1/assignment_loss‚4.589960185686747 -loss/FRt2/detection_loss‚2.12176880430668 1loss/SRqqt2/detection_loˆÆûšÅss‚1.4324813328865098 $ _timestamp‚1.788410375374166e+09 _runtime‚2985.786217733 1loss/SRqqt1/assignment_loss‚2.627455691839087 /loss/FRt2/assignment_loss‚2.785663751845664 1loss/SRqqt2/assignment_loss‚1.700154625606246 2loss/SRqqt3/assignment_loss‚1.8486149351422985 2loss/SRqqt4/assignment_loss‚1.8884610907976018 1loss/FBt4/assignment_loss‚0.46028379701101585 0loss/SRqqt3/detection_loss‚1.801029102333511 1loss/SRqqt4/detection_loss‚2.0778950612212337 /loss/FBt3/detection_loss‚0.7513084221401567 -loss/FRt4/detection_loss‚2.16330669231193 /loss/FBt1/detection_loss‚3.1034642566334116 /loss/FBt2/detection_loss‚3.3620307530675615 epoch‚3È´ÖëÅ‘Þ‚(:† •ôãÔãé™ gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.654' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v´Ì¨&ß‚(:› •ôãÔü‹šœ disk.rbd2.out‚70.19091796875 disk.sda3.out‚ 129.00390625" proc.memory.rssMB‚ 2798.7578125 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 710.40625( disk.nvme0c0n1.out‚50217.90966796875 network.sent‚ 8.646614e+06) disk./.usagePercent‚73.29704121033153 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 1.595703125 disk.dm-1.in‚0) proc.memory.percent‚17.08226203918457 disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda.out‚ 129.00390625$ disk./.usageGB‚1092.178882598877 disk.sda.in‚0 disk.sda1.out‚0# disk.dm-0.out‚50217.90966796875$ memory_percent‚98.62148761749268 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 710.40625 disk.dm-1.out‚ 129.00390625& disk.nvme0n1.out‚50217.90966796875( proc.memory.availableMB‚ 225.85546875 cpu‚25.37454269080276 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 710.40625 proc.cpu.threads‚17 disk.rbd3.in‚0 network.recv‚ 1.114118e+06 disk.rbd0.out‚0œP¢à‚(:„ ¤ôãÔ˺£™ gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.48' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.896 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07é.Yá‚(:“ ¤ôãÔ‘÷ë› disk.rbd0.in‚0 disk.sda.out‚ 129.55859375" proc.memory.rssMB‚ 2798.7578125 cpu‚25.373029681948374 disk.rbd0.out‚0 disk.nvme0n1.in‚ 710.40625$ memory_percent‚98.62148761749268$ disk./.usageGB‚1092.178882598877 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 710.40625( proc.memory.availableMB‚ 225.85546875 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 129.55859375 disk.drbd1777.in‚0$ disk.nvme0n1.out‚50934.134765625 network.recv‚ 1.120727e+06& disk.nvme0c0n1.out‚50934.134765625 disk.sda2.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0) proc.memory.percent‚17.08226203918457 disk.rbd1.out‚ 1.595703125 disk.dm-1.out‚ 129.55859375! disk.dm-0.out‚50934.134765625 network.sent‚ 8.754728e+06) disk./.usagePercent‚73.29704121033153 disk.rbd2.out‚ 70.27734375 disk.sda1.out‚0 disk.rbd1.in‚0ayÛK2â- _wandbruntime‚3020  _runtime‚3020Æô1dÆãÀ $ _timestamp‚1.788410409688057e+09 _runtime‚3020.100233655 _step‚329 & lr-AdamW/pg1‚8.498242837653792e-05 & lr-AdamW/pg2‚8.498242837653792e-05 trainer/global_step‚4049V µ™ËäÅ $ _timestamp‚1.788410409688057e+09 _runtime‚3020.100233655 _step‚329 & lr-AdamW/pg1‚8.498242837653792e-05 & lr-AdamW/pg2‚8.498242837653792e-05 trainer/global_step‚4049Éš¸±Q2å- _wandbruntime‚3020  _runtime‚3020(0²,Û'æÕO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31113556 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚108463142028288 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚119727772516864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81805786 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5808483768320 _step‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚114019110502400 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81825824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23015271 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53302285 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚120813727239168 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119727772516864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135128109 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54128825 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚561 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5708849017344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚114155373739150 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚126621974385664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31113554 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81805786 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54129388 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53302283 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚114019092499456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5708849017344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5708680017408 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚120813703299072±Xó(à'çÚO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚114155555899914 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚119727959519744 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119727772516864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135128109 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54128825 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚561 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5708849017344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚114155373739150 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚126621974385664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31113554 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81805786 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54129388 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53302283 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚114019092499456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5708849017344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5708680017408 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚120813703299072 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31113556 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚108463142028288 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚119727772516864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81805786 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5808483768320 _step‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚114019110502400 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81825824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23015271 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53302285 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚120813727239168 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 _runtime‚3020.741610346 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135140515 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119756435019776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚114046379867136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53314942 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54141757 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚114182888247002 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5809903592448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126652311365632 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚114182706086030 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81825573 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚114046361864192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23020826 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚119756622035968 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5710242168832 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53314944 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135160581 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚114046379867136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120842616247296 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5710242168832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135140515 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31120931 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚108489085210624 % _timestamp‚1.7884104103287454e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5710073155584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53314944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120842640187392 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5693620875406 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5693785997018 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199Ƴœê&êäM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54142283 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81845637 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81825573 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 _runtime‚3020.741610346 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135140515 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119756435019776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚114046379867136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53314942 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54141757 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚114182888247002 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5809903592448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126652311365632 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚114182706086030 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81825573 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚114046361864192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23020826 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚119756622035968 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5710242168832 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53314944 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135160581 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚114046379867136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120842616247296 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5710242168832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135140515 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31120931 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚108489085210624 % _timestamp‚1.7884104103287454e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5710073155584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53314944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120842640187392 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5693620875406 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5693785997018 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81845637 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230330368 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚115522004674358 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5761703529270 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚121160912076288 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚121161099079168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5761868650674 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54768666 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5879143693824 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚109760301145088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚115382578739200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31482404 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5778351340032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚3054.818186984 _step‚334 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206390272 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚115382578739200 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚527 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7884104444057906e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230330368 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚115522004674358 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5761703529270 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚121160912076288 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚121161099079168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5761868650674 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54768666 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5879143693824 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚109760301145088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚115382578739200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31482404 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 WEDeviceStatsMonitor.on_µG4ìtrain_batch_start/active_bytes.small_pool.freed‚ 5778351340032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚3054.818186984 _step‚334 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206390272 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚115382578739200 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚527 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7884104444057906e+09 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82818275 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚121189761595392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136786233 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23291566 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5763092693814 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚3055.466864006 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82818275 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5779913491456 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚128169055493120 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31489783 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 % _timestamp‚1.7884104450540636e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82838339 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚121189761595392 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136766167 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 trainer/global_step‚4099 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚115409848103936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚128168823078912 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚109786261366784 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚109786244327424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136766167 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚115409830100992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚121189574579200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54781347 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53947892 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53947894 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115409848103936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5779744478208 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚115549337021238 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82818275 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚121189761595392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136786233 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23291566 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5763092693814 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚3055.466864006 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82818275 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5779913491456 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚128169055493120 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31489783 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 % _timestamp‚1.7884104450540636e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82838339 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚121189761595392 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136766167 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 trainer/global_step‚4099 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚115409848103936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚128168823078912 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚109786261366784 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚109786244327424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136766167 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚115409830100992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚121189574579200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54781347 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53947892 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53947894 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115409848103936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5779744478208 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚115549337021238 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 B‚÷¹"%d_bytes.small_pool.freed‚ 5831175347678 _runtime‚3089.514792732 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚116888817770330 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚122594051635712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55409666 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚116888635609566 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5848191662592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚122594238638592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚555 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23558412 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5950169623552 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4149 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚122594238638592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5848022662656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54568183 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚111057460261888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83811228 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5831340469082 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416'`Š‹à'‡ÚO N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚116888635609566 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5848191662592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚122594238638592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚555 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23558412 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5950169623552 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4149 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚122594238638592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5848022662656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54568183 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚111057460261888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83811228 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5831340469082 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138359373 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚116746028973056 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83791190 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83791190 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5949959038976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5831175347678 _runtime‚3089.514792732 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚116888817770330 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚122594051635712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55409666Ò`"‘ˆ‚(:† ïôãÔï–•™ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.398' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è F!‰‚(:Š ïôãÔ›Þ½› disk.sda3.out‚ 132.296875 disk.drbd1777.out‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 225.4765625 disk.dm-0.in‚ 710.40625 disk.dm-0.out‚53078.47265625 disk.dm-1.out‚ 132.296875 disk.sda.in‚0$ disk./.usageGB‚1096.138916015625 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.out‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 710.40625$ memory_percent‚98.62380027770996 disk.sda2.in‚0* proc.memory.percent‚17.082571983337402 cpu‚25.365177021584536 network.sent‚ 8.989135e+06) disk./.usagePercent‚73.56280237561884# disk.nvme0n1.out‚53078.47265625 disk.rbd2.in‚0 proc.cpu.threads‚17 network.recv‚ 1.147278e+06 disk.rbd3.out‚0% disk.nvme0c0n1.out‚53078.47265625 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 132.296875 disk.drbd1777.in‚0 disk.rbd1.out‚ 1.744140625# proc.memory.rssMB‚ 2798.80859375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd2.out‚ 72.0859375qo÷2Š- _wandbruntime‚3090  _runtime‚3090ž× ¥å&‹ßM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚116773298337792 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚122622901154816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83831041 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5832564512222 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54580842 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚129685288367104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54580844 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31858633 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _runtime‚3090.159719948 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5849415800832 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83831041 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚116916150117418 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚116915967956446 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚116773298337792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚122622714138624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55422399 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5832729633834 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138391819 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83810977 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚123733935009792 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54580842 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23564301 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5951377297408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23563768 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138391819 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138411885 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚123733911069696 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138411885 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83810977 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ˆ„:‹ŒäM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 trainer/global_step‚4149 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚116773316340736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚116773316340736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31858631 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚111083403444224 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚122622901154816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5849415800832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129685522878464 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@D÷—Ã'Ú eviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 _step‚339 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111083420483584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _runtime‚3090.159719948 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5849415800832 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83831041 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚116916150117418 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚116915967956446 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚116773298337792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚122622714138624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55422399 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5832729633834 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138391819 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83810977 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚123733935009792 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54580842 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23564301 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5951377297408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23563768 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138391819 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138411885 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚123733911069696 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138411885 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83810977 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚116773298337792 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚122622901154816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83831041 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5832564512222 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54580842 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚129685288367104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54580844 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31858633 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚131171635760128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5917862985216 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5917693985280 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112354636418048 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚124027378198016 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6020757799936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84783892 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚131171403332608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32220106 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚118255266544774 % _timestamp‚1.7884105165173094e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56049579 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚124027191195136 trainer/global_step‚4199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55201133 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55201135 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 _runtime‚3126.930112399 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55201133 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5900812287490 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚505 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚124027191195136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5917693985280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚125150669472768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84803930 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚118255448705538 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23829473 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚124027378198016 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140005065 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6020966287360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56049072 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚118109497209856 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84783892 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _step‚342 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32220104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚118109497209856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5900647166086 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416ìÂïÒ´™ÚO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚118109515212800 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125150645532672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚131171635760128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5917862985216 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5917693985280 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112354636418048 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚124027378198016 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6020757799936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84783892 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚131171403332608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32220106 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚118255266544774 % _timestamp‚1.7884105165173094e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56049579 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚124027191195136 trainer/global_step‚4199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55201133 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55201135 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 _runtime‚3126.930112399 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55201133 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5900812287490 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚505 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚124027191195136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5917693985280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚125150669472768 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84803930 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚118255448705538 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23829473 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚124027378198016 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140005065 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6020966287360 F@DeviceStatsMonitor.on_train_batch_s×âS, tart/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56049072 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚118109497209856 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84783892 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _step‚342 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32220104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚118109497209856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5900647166086 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5917862985216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23828968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55201135 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460Ö•ÜšÃ2š- _wandbruntime‚3127  _runtime‚3127vdîÐå&›ßM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚131201968564736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32227481 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56061923 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚131201736150528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 % _timestamp‚1.7884105172086508e+09 _step‚343 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5919256136704 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125179558480896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55213794 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚118136766574592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚118136766574592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56062449 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140037537 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚125179582420992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚118282598891654 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55213792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6022386143744 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚124056040714240 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚124055853698048 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55213794 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6022177669632 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140037537 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚118136784577536 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 trainer/global_step‚4199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚112380579600384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23834442 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚3127.621621148òÆñqê&œäM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55213794 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6022177669632 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140037537 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚118136784577536 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 trainer/global_step‚4199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚112380579600384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23834442 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚3127.621621148 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚131201968564736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32227481 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56061923 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚131201736150528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 % _timestamp‚1.7884105172086508e+09 _step‚343 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5919256136704 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125179558480896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55213794 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚118136766574592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚118136766574592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56062449 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140037537 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚125179582420992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚118282598891654 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55213792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6022386143744 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚124056040714240 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚124055853698048 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257×nðÙ2- _wandbruntime‚3127  _runtime‚3127Ú§æMWžÑ /loss/FRt3/assignment_loss‚2.461059590180715 2loss/SRqqt4/assignment_loss‚1.9060984560269483 0loss/FBt4/assignment_loss‚0.4379231646312941 1loss/SRqqt4/detection_loss‚2.0075106474818014 %loss/total_loss‚51.25952284613163 _step‚344 .loss/FRt1/assignment_loss‚4.80705714225769 /loss/FRt3/detection_loss‚2.1905551659755216 .loss/FBt2/detection_loss‚3.623337456158229 trainer/global_step‚4199 .loss/FRt1/detection_loss‚2.911976698906191 1loss/SRqqt1/detection_loss‚2.4758778549730778 % _timestamp‚1.7884105172111454e+09 2loss/SRqqt3/assignment_loss‚1.9689657262836893 /loss/FRt2/detection_loss‚2.4911909240942736 /loss/FBt1/detection_loss‚2.9634670416514077 /loss/FRt4/assignment_loss‚4.173183013333214 .loss/FRt4/detection_loss‚2.329543350140254 epoch‚3 /loss/FRt2/assignment_loss‚3.235729252680754 1loss/SRqqt1/assignment_loss‚2.505883054807782 0loss/FBt1/assignment_loss‚0.5093820030483585 1loss/FBt3/assignment_loss‚0.29874834644281545 /loss/FBt4/detection_loss‚2.5226321731294905 2loss/SRqqt2/assignment_loss‚1.4490391830530236 0loss/SRqqt3/detection_loss‚1.499187961841623 1loss/FBt2/assignment_loss‚0.47275392711162567 1loss/SRqqt2/detection_loss‚1.3750950403073254 /loss/FBt3/detection_loss‚0.6433256716145209 _runtime‚3127.623300683rd<Ì\ŸÖ /loss/FBt3/detection_loss‚0.6433256716145209 _runtime‚3127.623300683 /loss/FRt3/assignment_loss‚2.461059590180715 2loss/SRqqt4/assignment_loss‚1.9060984560269483 0loss/FBt4/assignment_loss‚0.4379231646312941 1loss/SRqqt4/detection_loss‚2.0075106474818014 %loss/total_loss‚51.25952284613163 _step‚344 .loss/FRt1/assignment_loss‚4.80705714225769 /loss/FRt3/detection_loss‚2.1905551659755216 .loss/FBt2/detection_loss‚3.623337456158229 trainer/global_step‚4199 .loss/FRt1/detection_loss‚2.911976698906191 1loss/SRqqt1/detection_loss‚2.4758778549730778 % _timestamp‚1.7884105172111454e+09 2loss/SRqqt3/assignment_loss‚1.9689657262836893 /loss/FRt2/detection_loss‚2.4911909240942736 /loss/FBt1/detection_loss‚2.9634670416514077 /loss/FRt4/assignment_loss‚4.173183013333214 .loss/FRt4/detection_loss‚2.329543350140254 epoch‚3 /loss/FRt2/assignment_loss‚3.235729252680754 1loss/SRqqt1/assignment_loss‚2.505883054807782 0loss/FBt1/assignment_loss‚0.5093820030483585 1loss/FBt3/assignment_loss‚0.29874834644281545 /loss/FBt4/detection_loss‚2.5226321731294905 2loss/SRqqt2/assignment_loss‚1.4490391830530236 0loss/SRqqt3/detection_loss‚1.499187961841623 1loss/FBt2/assignment_loss‚0.47275392711162567 1loss/SRqqt2/detection_loss‚1.3750950403073254ØÏ?èç‘ ‚(:† œõãÔ¾¥˜™ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.188' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4376 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êé•â(¡‚(: œõãÔùèò› disk.rbd3.in‚0 network.recv‚ 1.166073e+06# disk./.usageGB‚1096.26859664917 disk.sda2.in‚0 disk.sda.out‚ 133.94140625# proc.memory.rssMB‚ 2798.80859375 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.dm-1.out‚ 133.94140625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 710.40625 disk.drbd1777.out‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.rbd1.out‚ 1.7763671875 disk.rbd2.out‚72.37548828125 disk.rbd3.out‚0( disk.nvme0c0n1.out‚55010.06787109375 cpu‚25.360681242150257 disk.sda2.out‚0 disk.dm-0.in‚ 710.40625 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 710.40625 disk.dm-1.in‚0* proc.memory.percent‚17.082571983337402) disk./.usagePercent‚73.57150535174533 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda1.out‚0 network.sent‚ 9.190253e+06 disk.sda3.out‚ 133.94140625& disk.nvme0n1.out‚55010.06787109375# disk.dm-0.out‚55010.06787109375$ memory_percent‚98.62303733825684' proc.memory.availableMB‚ 225.6015625,o£‘¢‚(:† «õãÔ›‚Á™ gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.811' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5622 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”›Ðª)£‚(:ž «õãÔˆÕÿ› disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda.out‚ 134.56640625 disk.rbd1.in‚0 disk.rbd2.out‚73.96533203125 disk.sda.in‚0 disk.drbd1777.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 225.33984375 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 1.83935546875# proc.memory.rssMB‚ 2798.80859375 cpu‚25.359211727844887 proc.cpu.threads‚17$ disk./.usageGB‚1096.268669128418 disk.dm-0.in‚ 710.40625 disk.rbd3.out‚0* proc.memory.percent‚17.082571983337402 disk.dm-1.out‚ 134.56640625( disk.nvme0c0n1.out‚55010.34912109375 network.sent‚ 9.29856e+06 disk.nvme0c0n1.in‚ 710.40625 disk.sda3.out‚ 134.56640625 disk.sda1.in‚0 disk.drbd1777.in‚0$ memory_percent‚98.62463474273682# disk.dm-0.out‚55010.34912109375 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda1.out‚0 network.recv‚ 1.172388e+06( disk./.usagePercent‚73.5715102158885 disk.nvme0n1.in‚ 710.40625& disk.nvme0n1.out‚55010.34912109375ŠØ‹¨2¤- _wandbruntime‚3162  _runtime‚3162é FãÅ¥¿ % lr-AdamW/pg1‚8.27933944396971e-05 % lr-AdamW/pg2‚8.27933944396971e-05 trainer/global_step‚4249 % _timestamp‚1.7884105518197167e+09 _runtime‚3162.231856727 _step‚345fÒ0Ê¦Ä % lr-AdamW/pg2‚8.27933944396971e-05 trainer/global_step‚4249 % _timestamp‚1.7884105518197167e+09 _runtime‚3162.231856727 _step‚345 % lr-AdamW/pg1‚8.27933944396971e-05Ùþ¯â2§- _wandbruntime‚3162  _runtime‚3162÷^»ô¨ÔO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55834085 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5987534307840 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚119472965446656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85796632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current€-†Â‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55834085 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚125460330754560 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚125460517757440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚113651795534848 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚119472983449600 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚119472983449600 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85776594 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚132688169913856 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24100973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141610677 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85796632 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚126596316883968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6091853029888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚4249 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5970284105898 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5987365307904 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85776594 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚132687929097728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55834083 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56689927 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24101503 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚08—Éß'©ÙO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚132688169913856 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24100973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141610677 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85796632 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚126596316883968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6091853029888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚4249 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5970284105898 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5987365307904 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85776594 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚132687929097728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55834083 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56689927 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24101503 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55834085 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5987534307840 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚119472965446656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85796632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55834085 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚125460330754560 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚125460517757440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚113651795534848 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚119472983449600 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚119472983449600 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85776594 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248Ú|[^œ2ª- _wandbruntime‚3162  _runtime‚3162«stå&«ßM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5988927459328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141663189 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5988927459328 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55846744 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚119500234811392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119649229826862 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚119500252814336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85796381 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚125489180273664 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚119500252814336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85816445 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141643123 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56702724 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85816445 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24106393 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141643123 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32596333 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4249 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5988758446080 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚119500234811392 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6093055465472úW‰I%¬äM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚125488993257472 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚132718495868928 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚126625229832192 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85796381 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚113677738717184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚119500252814336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85816445 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141643123 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56702724 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85816445 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24106393 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141643123 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32596333 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4249 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5988758446080 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚119500234811392 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6093055465472 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5988927459328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141663189 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5988927459328 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55846744 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚119500234811392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119649229826862 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚119500252814336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85796381 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚125489180273664 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 F¹Ê¸Ä % _timestamp‚1.7884105870299547e+09 _runtime‚3197.442135672 _step‚349 % lr-AdamW/pg1‚8.22290315386814e-05 % lr-AdamW/pg2‚8.22290315386814e-05 trainer/global_step‚4299ÝGÞ2¹- _wandbruntime‚3197  _runtime‚3197Ó;%ñÚ'ºÔO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24371507 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚126893470313984 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚126893470313984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 % _timestamp‚1.7884105870306916e+09 _step‚350 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57329855 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128041964295168 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6057205630464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚126893657316864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6057036630528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86769296 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24371507 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚126893470313984 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚126893470313984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 % _timestamp‚1.7884105870306916e+09 _step‚350 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57329855 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128041964295168 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6057205630464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚126893657316864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6057036630528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86769296 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚126922319833088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚499 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134235000930304 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚114974897833984 _runtime‚3198.095441281 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6058598781952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚121015860762070 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚114974880794624 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86789083 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6164123686912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143288841 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128070853303296 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143288841 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚126922319833088 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚120863721051136 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56479692 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57342220 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 143268775 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134234772710400 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86809147 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32965183 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56479694 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚120863721051136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6058598781952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128070877243392 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6163919407104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0Ä+~ê&¾äM trainer/global_step‚4299 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57342220 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 143268775 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134234772710400 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86809147 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32965183 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56479694 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚120863721051136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6058598781952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128070877243392 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6163919407104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚121016042923042 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚351 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6058429768704 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32965181 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24377538 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56479694 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57342721 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 % _timestamp‚1.7884105876827102e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚126922319833088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚499 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134235000930304 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚114974897833984 _runtime‚3198.095441281 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6058598781952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚121015860762070 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚114974880794624 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86789083 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6164123686912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143288841 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128070853303296 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143288841 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚126922319833088 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚120863721051136 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56479692 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚128326609873408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚122199919923200 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚122199901920256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87782036 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57099983 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚122199919923200 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚128326609873408 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6126707953152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚4349 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87782036 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6233585167360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87761998 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33326656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚129487587766272 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144861981 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚116246113768448 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57099983 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚128326609873408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚122199919923200 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚122199901920256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87782036 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57099983 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚122199919923200 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚128326609873408 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6126707953152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚4349 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87782036 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6233585167360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87761998 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33326656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚129487587766272 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144861981 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚116246113768448 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57099983 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚355 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57981485 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57980967 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6110616907466 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33334033 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚122382673858250 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 % _timestamp‚1.7884106224124281e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚122227171284992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6128270104576 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24646936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6110451785854 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87801849 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6234792247808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚128355272376320 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚122227171284992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144894427 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144914493 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33334031 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚135751292962304 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚116272039911424 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57112642 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚122382491697278 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144914493 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144894427 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87781785 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚116272056950784 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚122227189287936%îÌ|&ÎäM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚128355272376320 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚122227171284992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144894427 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144914493 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33334031 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚135751292962304 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚116272039911424 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57112642 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚122382491697278 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144914493 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144894427 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87781785 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚116272056950784 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚122227189287936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚355 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57981485 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57980967 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6110616907466 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33334033 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚122382673858250 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 % _timestamp‚1.7884106224124281e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚122227171284992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6128270104576 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24646936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6110451785854 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87801849 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ëÈ#Gn 218959872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6234792247808ãgým2Ï- _wandbruntime‚3232  _runtime‚3232´j9ÆZÐÔ /loss/FRt4/detection_loss‚2.5660584186730175 trainer/global_step‚4349 2loss/SRqqt3/assignment_loss‚2.1461718872837396 1loss/SRqqt1/detection_loss‚3.0236141213348935 2loss/FBt1/assignment_loss‚0.031190632564539557 0loss/FBt2/assignment_loss‚0.4872604489326477 0loss/FBt3/assignment_loss‚0.2352236558966213 /loss/FBt3/detection_loss‚0.7997986508876478 /loss/FRt4/assignment_loss‚3.867605774298958 /loss/FRt2/detection_loss‚2.0795205998838995 /loss/SRqqt4/detection_loss‚1.80614347726691 2loss/SRqqt4/assignment_loss‚1.6225842662593897 0loss/FBt4/assignment_loss‚0.6583579199959786 1loss/SRqqt2/detection_loss‚1.5684354883432388 /loss/FBt2/detection_loss‚3.3692699551582335 _runtime‚3232.826654014 /loss/FRt2/assignment_loss‚3.100232452677007 0loss/FRt3/assignment_loss‚2.6580374108420477 1loss/SRqqt1/assignment_loss‚2.788017267272586 2loss/SRqqt2/assignment_loss‚1.6314957475165526 .loss/FRt3/detection_loss‚1.968812929259406 /loss/FBt4/detection_loss‚2.1332449670881033 _step‚356 /loss/FRt1/assignment_loss‚5.143811146418254 /loss/FRt1/detection_loss‚2.8853212161497637 %loss/total_loss‚53.55215928525792 epoch‚3 % _timestamp‚1.7884106224145439e+09 1loss/SRqqt3/detection_loss‚1.4369006355333587 /loss/FBt1/detection_loss‚5.5450502157211305&ÄÆ_ÑÙ .loss/FRt3/detection_loss‚1.968812929259406 /loss/FBt4/detection_loss‚2.1332449670881033 _step‚356 /loss/FRt1/assignment_loss‚5.143811146418254 /loss/FRt1/detection_loss‚2.8853212161497637 %loss/total_loss‚53.55215928525792 epoch‚3 % _timestamp‚1.7884106224145439e+09 1loss/SRqqt3/detection_loss‚1.4369006355333587 /loss/FBt1/detection_loss‚5.5450502157211305 /loss/FRt4/detection_loss‚2.5660584186730175 trainer/global_step‚4349 2loss/SRqqt3/assignment_loss‚2.1461718872837396 1loss/SRqqt1/detection_loss‚3.0236141213348935 2loss/FBt1/assignment_loss‚0.031190632564539557 0loss/FBt2/assignment_loss‚0.4872604489326477 0loss/FBt3/assignment_loss‚0.2352236558966213 /loss/FBt3/detection_loss‚0.7997986508876478 /loss/FRt4/assignment_loss‚3.867605774298958 /loss/FRt2/detection_loss‚2.0795205998838995 /loss/SRqqt4/detection_loss‚1.80614347726691 2loss/SRqqt4/assignment_loss‚1.6225842662593897 0loss/FBt4/assignment_loss‚0.6583579199959786 1loss/SRqqt2/detection_loss‚1.5684354883432388 /loss/FBt2/detection_loss‚3.3692699551582335 _runtime‚3232.826654014 /loss/FRt2/assignment_loss‚3.100232452677007 0loss/FRt3/assignment_loss‚2.6580374108420477 1loss/SRqqt1/assignment_loss‚2.788017267272586 2loss/SRqqt2/assignment_loss‚1.6314957475165526äßõþ‘Ò‚(:† …öãÔà¡Â™ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚104.861' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.9722 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]jvÓ‚(:Ž …öãÔòúÂœ disk.sda2.in‚0 disk.sda3.out‚ 137.7578125 disk.sda.out‚ 137.7578125 disk.drbd1777.in‚0 disk.rbd3.out‚0% disk.nvme0c0n1.out‚56316.22265625# disk.nvme0n1.out‚56316.22265625 disk.rbd2.out‚ 75.7939453125( proc.memory.availableMB‚ 224.66015625 cpu‚25.35067860429893 proc.cpu.threads‚17 network.recv‚ 1.203549e+06 disk.sda.in‚0 disk.nvme0n1.in‚ 710.40625$ memory_percent‚98.62878322601318# proc.memory.rssMB‚ 2798.80859375 network.sent‚ 9.539626e+06 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0( disk./.usagePercent‚73.6098527202992$ disk./.usageGB‚1096.840000152588 disk.sda3.in‚0 disk.rbd1.out‚ 1.90234375 disk.dm-0.out‚56316.22265625 disk.dm-1.out‚ 137.7578125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 710.40625 disk.rbd2.in‚0* proc.memory.percent‚17.082571983337402 disk.sda1.out‚0 disk.rbd3.in‚0}—@ë‘Ô‚(:† ”öãÔÛ¹¬™ gpu.0.gpu‚20.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.936' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1872 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03‰Æ”(Õ‚(: ”öãÔµÐÈœ disk.sda1.in‚0 disk.drbd1777.out‚0 disk.rbd2.out‚ 75.8564453125$ memory_percent‚98.62873554229736 cpu‚25.349302786325644 disk.dm-1.out‚ 138.37109375 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda.out‚ 138.37109375 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 710.40625 disk.nvme0c0n1.in‚ 710.40625 disk.sda3.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17 network.recv‚ 1.209708e+06 disk.rbd1.out‚ 1.90234375 disk.rbd0.in‚0( proc.memory.availableMB‚ 224.66796875* proc.memory.percent‚17.082571983337402) disk./.usagePercent‚73.62654927567482& disk.nvme0n1.out‚56415.73095703125 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2798.80859375( disk.nvme0c0n1.out‚56415.73095703125% disk./.usageGB‚1097.0887908935547 disk.nvme0n1.in‚ 710.40625# disk.dm-0.out‚56415.73095703125 disk.dm-1.in‚0 disk.sda2.in‚0 network.sent‚ 9.647941e+06 disk.sda3.out‚ 138.37109375•®Ip2Ö- _wandbruntime‚3267  _runtime‚3267©¹ÁÇÇ×Á & lr-AdamW/pg1‚8.108059874904143e-05 & lr-AdamW/pg2‚8.108059874904143e-05 trainer/global_step‚4399 % _timestamp‚1.7884106570478137e+09 _runtime‚3267.459927923 _step‚357´Í§ÌØÆ _runtime‚3267.459927923 _step‚357 & lr-AdamW/pg1‚8.108059874904143e-05 & lr-AdamW/pg2‚8.108059874904143e-05 trainer/global_step‚4399 % _timestamp‚1.7884106570478137e+09åpÄË2Ù- _wandbruntime‚3267  _runtime‚32678”Ñ©Û'ÚÕO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚123563388160000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58608557 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130933259117568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6196548275712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24913051 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6178534439718 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚123563370157056 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129759936435712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6304260658688 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚123721972446370 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88754700 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129759749432832 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146507673 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚137237495836160 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6196379275776 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 _runtime‚3267.461073361 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884106570486195e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57732935 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88774738 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6178699561122 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚358 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57732935 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33695504 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚130933235177472 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57732933 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6196548275712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24913051 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6178534439718 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚123563370157056 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129759936435712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6304260658688 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚123721972446370 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88754700 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129759749432832 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146507673 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚137237495836160 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6196379275776 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 _runtime‚3267.461073361 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884106570486195e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57732935 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248æ\öOø2Ü- _wandbruntime‚3268  _runtime‚3268ò—¼ÇðÝÝM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚137268060822528 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146540145 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88774487 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6305678185472 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88774487 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117569199028224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6197772413952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58620703 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚130962148125696 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57745594 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚123590639521792 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6197772413952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚123590639521792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57745592 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6197941427200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚123749304793458 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146540145 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_en¶[:¾ód/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6197941427200 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚137267826311168 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24917822 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88794551 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 $ _timestamp‚1.788410657711808e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚123590657524736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6305888756736 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚ 3268.12465587 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129788411935744 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58621245 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146520079 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚130962172065792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚129788598951936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542DS»¤è&ÞâM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 $ _timestamp‚1.788410657711808e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚123590657524736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6305888756736 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚ 3268.12465587 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129788411935744 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58621245 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146520079 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚130962172065792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚129788598951936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚137268060822528 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146540145 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88774487 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6305678185472 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88774487 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117569199028224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6197772413952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58620703 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚130962148125696 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57745594 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚123590639521792 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6197772413952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚123590639521792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57745592 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6197941427200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚123749304793458 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146540145 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6197941427200 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚137267826311168 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24917822 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88794551 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ç(Ö²2ß- _wandbruntime‚3268  _runtime‚3268=MzSàÍ 1loss/SRqqt1/detection_loss‚3.0009789081911245 1loss/SRqqt3/detection_loss‚1.5731026031533066 2loss/SRqqt1/assignment_loss‚2.2030352192620435 .loss/FRt1/detection_loss‚2.826772411664327 .loss/FRt3/detection_loss‚2.317160614627473 trainer/global_step‚4399 _step‚360 /loss/FRt1/assignment_loss‚4.896917192141215 /loss/FRt2/detection_loss‚1.8374906973760636 /loss/FBt1/detection_loss‚3.3490301072597504 /loss/FBt4/detection_loss‚2.3038596969384413 %loss/total_loss‚51.69597433418092 2loss/SRqqt4/assignment_loss‚2.1983106472390763 /loss/SRqqt2/detection_loss‚1.31172459280457 /loss/FRt2/assignment_loss‚3.098320170504148 /loss/FRt3/assignment_loss‚2.860714561127602 /loss/FRt4/assignment_loss‚3.804836799701055 /loss/FBt2/detection_loss‚2.8704337239265443 0loss/FBt1/assignment_loss‚0.6983729199739173 0loss/FBt2/assignment_loss‚0.5050187520682812 0loss/FBt3/assignment_loss‚0.2977077538859305 /loss/FRt4/detection_loss‚2.4475815991560617 .loss/FBt3/detection_loss‚1.021205610510422 $ _timestamp‚1.788410657713952e+09 _runtime‚3268.125981748 2loss/SRqqt2/assignment_loss‚1.5760328793813558 2loss/SRqqt3/assignment_loss‚1.9551788191406094 0loss/FBt4/assignment_loss‚0.8724017326648419 /loss/SRqqt4/detection_loss‚1.86978632148276 epoch‚3ЃxXáÒ /loss/FRt4/detection_loss‚2.4475815991560617 .loss/FBt3/detection_loss‚1.021205610510422 $ _timestamp‚1.788410657713952e+09 _runtime‚3268.125981748 2loss/SRqqt2/assignment_loss‚1.5760328793813558 2loss/SRqqt3/assignment_loss‚1.9551788191406094 0loss/FBt4/assignment_loss‚0.8724017326648419 /loss/SRqqt4/detection_loss‚1.86978632148276 epoch‚3 1loss/SRqqt1/detection_loss‚3.0009789081911245 1loss/SRqqt3/detection_loss‚1.5731026031533066 2loss/SRqqt1/assignment_loss‚2.2030352192620435 .loss/FRt1/detection_loss‚2.826772411664327 .loss/FRt3/detection_loss‚2.317160614627473 trainer/global_step‚4399 _step‚360 /loss/FRt1/assignment_loss‚4.896917192141215 /loss/FRt2/detection_loss‚1.8374906973760636 /loss/FBt1/detection_loss‚3.3490301072597504 /loss/FBt4/detection_loss‚2.3038596969384413 %loss/total_loss‚51.69597433418092 2loss/SRqqt4/assignment_loss‚2.1983106472390763 /loss/SRqqt2/detection_loss‚1.31172459280457 /loss/FRt2/assignment_loss‚3.098320170504148 /loss/FRt3/assignment_loss‚2.860714561127602 /loss/FRt4/assignment_loss‚3.804836799701055 /loss/FBt2/detection_loss‚2.8704337239265443 0loss/FBt1/assignment_loss‚0.6983729199739173 0loss/FBt2/assignment_loss‚0.5050187520682812 0loss/FBt3/assignment_loss‚0.2977077538859305èÅ9C‘â‚(:† £öãÔßÀ°š gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚111.612' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3224 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<Þw%ã‚(:š £öãÔáªù› cpu‚25.347874084050172 disk.sda3.in‚0 disk.rbd1.out‚ 1.90234375# proc.memory.rssMB‚ 2798.80859375) disk./.usagePercent‚73.62665295872618 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1777.in‚0* proc.memory.percent‚17.082571983337402 network.recv‚ 1.213491e+06% disk./.usageGB‚1097.0903358459473 disk.nvme0n1.in‚ 710.40625 disk.dm-1.out‚ 139.01171875 disk.rbd2.out‚ 75.8720703125' disk.nvme0c0n1.out‚56416.1279296875 disk.sda.out‚ 139.01171875 disk.rbd1.in‚0 network.sent‚ 9.653466e+06 disk.dm-0.in‚ 710.40625 disk.rbd0.in‚0 disk.sda3.out‚ 139.01171875 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1777.out‚0$ memory_percent‚98.62802028656006% disk.nvme0n1.out‚56416.1279296875 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 224.78515625 proc.cpu.threads‚17" disk.dm-0.out‚56416.1279296875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda1.in‚0la5‘ä‚(:† ²öãԆʯ™ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.687' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1374 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g‚‡ñ#å‚(:˜ ²öãÔïöŒ disk.dm-0.in‚ 710.40625 disk.dm-1.in‚0# proc.memory.rssMB‚ 2798.80859375 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 710.40625# disk.dm-0.out‚56515.57568359375 disk.dm-1.out‚ 139.5546875 disk.sda3.in‚0 disk.sda2.out‚0 network.sent‚ 9.762595e+06 disk.rbd0.in‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚56515.57568359375 network.recv‚ 1.220367e+06 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd3.out‚0$ memory_percent‚98.62985610961914 disk.nvme0c0n1.in‚ 710.40625( disk./.usagePercent‚73.6266501426433% disk./.usageGB‚1097.0902938842773 disk.rbd2.out‚75.95849609375& proc.memory.availableMB‚ 224.484375 disk.sda3.out‚ 139.5546875* proc.memory.percent‚17.082571983337402 proc.cpu.threads‚17& disk.nvme0n1.out‚56515.57568359375 disk.sda.out‚ 139.5546875 disk.drbd1777.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 cpu‚25.346505829433774 disk.sda2.in‚0 disk.rbd1.out‚ 1.90234375p ±›æ‚(: ÁöãÔö¢’™ gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚98.647' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.729400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëÞ@ç‚(:û ÁöãÔ²ÈÂœ$ memory_percent‚98.62983226776123( proc.memory.availableMB‚ 224.48828125* proc.memory.percent‚17.082571983337402 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 710.40625 disk.sda1.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚140 disk.rbd0.out‚0& disk.nvme0c0n1.out‚56516.537109375 disk.sda2.out‚0$ disk.nvme0n1.out‚56516.537109375 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0 disk.rbd1.out‚ 1.90234375 cpu‚25.345177328307063 network.sent‚ 9.767755e+06 network.recv‚ 1.224198e+06' disk./.usagePercent‚73.626659102907! disk.dm-0.out‚56516.537109375 disk.rbd2.out‚75.97412109375 disk.sda1.out‚0 disk.rbd1.in‚0% disk./.usageGB‚1097.0904273986816 disk.sda3.out‚140 disk.drbd1777.in‚0 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2798.80859375 disk.sda.in‚0 disk.sda.out‚140 disk.dm-0.in‚ 710.40625 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.in‚0‚Ç@Û2è- _wandbruntime‚3302  _runtime‚3302D¥ #ÆéÀ $ _timestamp‚1.788410691996585e+09 _runtime‚3302.408712196 _step‚361 & lr-AdamW/pg1‚8.049676819208919e-05 & lr-AdamW/pg2‚8.049676819208919e-05 trainer/global_step‚4449î2KËêÅ & lr-AdamW/pg2‚8.049676819208919e-05 trainer/global_step‚4449 $ _timestamp‚1.788410691996585e+09 _runtime‚3302.408712196 _step‚361 & lr-AdamW/pg1‚8.049676819208919e-05éö°Ù‘2ë- _wandbruntime‚3302  _runtime‚3302Ýa1Û'ìÕO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58365885 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25181551 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚124926838393856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚131192888992256 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚132378882588672 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚118840414962688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131192888992256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚138753975355392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6248171379530 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚124926838393856 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148133325 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚138754214074368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59245907 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚132378906528768 _runtime‚3302.409655884 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89747402 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚527 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58365885 _step‚362 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148113285 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚529 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚131193075995136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6266219598336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚4449 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6375307545600 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25181024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58365883 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚118840432002048 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚125088603381578 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131192888992256 ZFDeviceStatsMonitor.on_train_batch_s_Ì<ê$tart/inactive_split_bytes.all.freed‚138753975355392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6248171379530 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚124926838393856 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148133325 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚138754214074368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59245907 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚132378906528768 _runtime‚3302.409655884 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89747402 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚527 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58365885 _step‚362 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148113285 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚529 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚131193075995136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6266219598336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚4449 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6375307545600 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25181024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58365883 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚118840432002048 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚125088603381578 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131221738511360 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59258066 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89767189 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚125115753567694 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚124954107758592 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚125115935728666 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148165797 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58378542 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6267612749824 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6267443736576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 148145731 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6376508458496 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚118866375184384 _step‚363 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 148145731 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚124954125761536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34071733 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148165797 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚138784303995392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _runtime‚3303.069972484 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89787253 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131221738511360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚132407795536896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6267612749824 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚138784542701056 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6376723224064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚4449 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58378544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25186335 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6249560544282 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34071731 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131221551495168 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚132407819476992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 $ _timestamp‚1.788410692657359e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664¥È‰®é&ðãM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58378544 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89767189 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚125115753567694 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚124954107758592 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚125115935728666 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148165797 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58378542 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6267612749824 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6267443736576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 148145731 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6376508458496 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚118866375184384 _step‚363 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 148145731 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚124954125761536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34071733 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148165797 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚138784303995392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _runtime‚3303.069972484 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89787253 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131221738511360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚132407795536896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6267612749824 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚138784542701056 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6376723224064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚4449 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58378544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25186335 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6249560544282 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34071731 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131221551495168 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚132407819476992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 $ _timestamp‚1.788410692657359e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6249395422670 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131221738511360 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59258066 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚132626028551680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6335721921024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25454324 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6335890920960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚126290324633600 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34433204 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6445979496448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚132626028551680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _runtime‚3337.603086534 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58998833 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90760142 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133824553939968 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58998835 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6446192178176 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90740104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚126290306630656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58998835 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚126290324633600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚120137574079488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚140270746118144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚132626215554560 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59887530 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133824529999872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34433206 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚126290306630656 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149758977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚132626028551680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6335721921024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25454324 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6335890920960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚126290324633600 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34433204 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6445979496448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚132626028551680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _runtime‚3337.603086534 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58998833 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90760142 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133824553939968 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58998835 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6446192178176 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90740104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚126290306630656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58998835 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚126290324633600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚120137574079488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚140270746118144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚132626215554560 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59887530 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133824529999872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34433206 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚126290306630656 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149758977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 J12þ- _wandbruntime‚3338  _runtime‚3338$°1&f$ÿßM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚126317575995392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59011494 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚126317593998336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚126482384502902 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6318867241078 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6337284072448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34440581 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140300841871872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚133853466888192 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59899692 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6447398923776 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90779955 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149771383 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6447609495040 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6337115059200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚120163517261824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34440583 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149791449 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25459629 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149791449 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_tra‘\8in_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6337284072448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34440581 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140300841871872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚133853466888192 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59899692 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6447398923776 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90779955 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149771383 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6447609495040 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6337115059200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚120163517261824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34440583 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149791449 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25459629 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149791449 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ïgí¼±2- _wandbruntime‚3338  _runtime‚3338F$æFR‚Ì /loss/FRt3/assignment_loss‚2.061133278452832 0loss/SRqqt1/detection_loss‚2.885373334089915 1loss/SRqqt4/detection_loss‚1.9407297466640119 $ _timestamp‚1.788410727850706e+09 _step‚368 2loss/SRqqt4/assignment_loss‚1.8827693029134362 1loss/FBt3/assignment_loss‚0.21957540570548817 /loss/FBt4/detection_loss‚2.2016656529158354 2loss/SRqqt3/assignment_loss‚1.6408821000903844 0loss/FBt2/assignment_loss‚0.1945736278874392 1loss/SRqqt2/detection_loss‚1.5223159357695513 0loss/SRqqt3/detection_loss‚1.445256199836731 /loss/FBt3/detection_loss‚0.8786828610134654 2loss/SRqqt1/assignment_loss‚2.3596637851248183 /loss/FRt2/detection_loss‚2.2040868723392486 ,loss/FBt1/detection_loss‚3.0401514342853 epoch‚3 0loss/FRt2/assignment_loss‚2.9794522964954377 0loss/FBt1/assignment_loss‚0.3156270925755962 .loss/FRt1/detection_loss‚2.912241551183885 /loss/FRt3/detection_loss‚2.1490526769472207 .loss/FRt1/assignment_loss‚4.98626410576605 0loss/FBt4/assignment_loss‚0.5049220057971979 %loss/total_loss‚49.69554373755923 /loss/FRt4/assignment_loss‚3.365637551413642 1loss/SRqqt2/assignment_loss‚1.557374146727609 .loss/FRt4/detection_loss‚2.281421351101663 trainer/global_step‚4499 .loss/FBt2/detection_loss‚4.166691422462463 _runtime‚3338.262757113qäe„WƒÑ 2loss/SRqqt3/assignment_loss‚1.6408821000903844 0loss/FBt2/assignment_loss‚0.1945736278874392 1loss/SRqqt2/detection_loss‚1.5223159357695513 0loss/SRqqt3/detection_loss‚1.445256199836731 /loss/FBt3/detection_loss‚0.8786828610134654 2loss/SRqqt1/assignment_loss‚2.3596637851248183 /loss/FRt2/detection_loss‚2.2040868723392486 ,loss/FBt1/detection_loss‚3.0401514342853 epoch‚3 0loss/FRt2/assignment_loss‚2.9794522964954377 0loss/FBt1/assignment_loss‚0.3156270925755962 .loss/FRt1/detection_loss‚2.912241551183885 /loss/FRt3/detection_loss‚2.1490526769472207 .loss/FRt1/assignment_loss‚4.98626410576605 0loss/FBt4/assignment_loss‚0.5049220057971979 %loss/total_loss‚49.69554373755923 /loss/FRt4/assignment_loss‚3.365637551413642 1loss/SRqqt2/assignment_loss‚1.557374146727609 .loss/FRt4/detection_loss‚2.281421351101663 trainer/global_step‚4499 .loss/FBt2/detection_loss‚4.166691422462463 _runtime‚3338.262757113 /loss/FRt3/assignment_loss‚2.061133278452832 0loss/SRqqt1/detection_loss‚2.885373334089915 1loss/SRqqt4/detection_loss‚1.9407297466640119 $ _timestamp‚1.788410727850706e+09 _step‚368 2loss/SRqqt4/assignment_loss‚1.8827693029134362 1loss/FBt3/assignment_loss‚0.21957540570548817 /loss/FBt4/detection_loss‚2.2016656529158354ðÀ!j‘„‚(:† îöãԵ“™ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.798' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7596 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e&OÅ$…‚(:™ îöãÔ„Ø› disk.rbd1.in‚0 disk.rbd3.in‚0* proc.memory.percent‚17.082571983337402 network.sent‚ 9.887518e+06( disk./.usagePercent‚73.6602134984048 disk.drbd1777.in‚0 disk.rbd2.out‚ 77.837890625 disk.sda.in‚0# disk.dm-0.out‚56715.29833984375 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0$ memory_percent‚98.63028526306152( disk.nvme0c0n1.out‚56715.29833984375# proc.memory.rssMB‚ 2798.80859375 network.recv‚ 1.239168e+06 disk.sda3.out‚ 141.66015625 disk.nvme0n1.in‚ 710.40625& disk.nvme0n1.out‚56715.29833984375 cpu‚25.34132239984511 disk.dm-0.in‚ 710.40625 disk.nvme0c0n1.in‚ 710.40625% disk./.usageGB‚1097.5904121398926 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.dm-1.out‚ 141.66015625 disk.rbd0.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 224.4140625 disk.sda3.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 1.90234375 disk.dm-1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 141.66015625ýÿT‘†‚(:† ýöãÔî»—™ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.756' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_±ò‡‚(: ýöãÔÒ‚ê›) disk./.usagePercent‚73.66021375441234 disk.drbd1777.out‚0 disk.sda.out‚ 142.2890625 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 710.40625 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.in‚0 network.recv‚ 1.245691e+06% disk./.usageGB‚1097.5904159545898 disk.sda1.in‚0 disk.rbd1.in‚0 disk.sda.in‚0& proc.memory.availableMB‚ 224.609375 network.sent‚ 9.996425e+06 disk.sda1.out‚0 disk.dm-1.out‚ 142.2890625 disk.rbd0.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2798.80859375 proc.cpu.threads‚17! disk.dm-0.out‚56814.759765625 disk.rbd1.out‚ 1.90234375 disk.rbd2.out‚ 77.853515625$ memory_percent‚98.62909317016602 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 142.2890625 disk.dm-0.in‚ 710.40625 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.rbd3.in‚0* proc.memory.percent‚17.082571983337402$ disk.nvme0n1.out‚56814.759765625& disk.nvme0c0n1.out‚56814.759765625 cpu‚25.33995526286704õ'V^2ˆ- _wandbruntime‚3372  _runtime‚3372²oäýljÁ _runtime‚3372.654385011 _step‚369 & lr-AdamW/pg1‚7.931048707562947e-05 & lr-AdamW/pg2‚7.931048707562947e-05 trainer/global_step‚4549 % _timestamp‚1.7884107622422159e+09&‚ëÜÌŠÆ & lr-AdamW/pg1‚7.931048707562947e-05 & lr-AdamW/pg2‚7.931048707562947e-05 trainer/global_step‚4549 % _timestamp‚1.7884107622422159e+09 _runtime‚3372.654385011 _step‚369ñPÏ2‹- _wandbruntime‚3372  _runtime‚3372ßdÜÛ'ŒÕO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚121434733196288 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚127653792870400 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91732806 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚507 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245010432 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚4549 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91732806 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚134059168111104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚127653792870400 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚121434750235648 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚134059168111104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6405393243648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚509 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6387115016346 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151384629 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚127821865251994 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25722264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6516804903936 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135270201351168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60524320 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34802056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚134059355113984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34802054 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 % _timestamp‚1.7884107622429295e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135270177411072 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127821683091230 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6386949894942 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚141786982315008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91752844 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚141787227325440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6517025974272 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59631783 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6405562243584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59631785 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151364589 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91752844 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25721757 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ª‰ ÚO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚121434733196288 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚127653792870400 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91732806 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚507 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245010432 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚4549 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91732806 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚134059168111104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚127653792870400 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚121434750235648 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚134059168111104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6405393243648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚509 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6387115016346 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151384629 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚127821865251994 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25722264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6516804903936 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135270201351168 ZLDeviceStatsMonióöÀtor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60524320 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34802056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚134059355113984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34802054 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 % _timestamp‚1.7884107622429295e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135270177411072 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127821683091230 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6386949894942 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚141786982315008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91752844 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚141787227325440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6517025974272 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59631783 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6405562243584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59631785 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151364589 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91752844 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25721757 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0òÍÂÙ2Ž- _wandbruntime‚3373  _runtime‚3373d­}gå&ßM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151397035 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59644442 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚127681044232192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60536888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60536369 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226122752 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6518231424512 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135299114299392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34809433 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151417101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141817547906560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7884107628949883e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6406786381824 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151397035 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚371 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91752593 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚134087830614016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25726938 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚134088017630208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59644442 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6388504181098 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚127681062235136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59644444 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚121460676378624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151417101 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6406955395072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127849015438110 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚127681062235136ÀâJê&äM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25727455 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6518433607168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202182656 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91772657 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91752593 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34809433 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151417101 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141817547906560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7884107628949883e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6406786381824 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151397035 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚371 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91752593 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚134087830614016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25726938 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚134088017630208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59644442 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6388504181098 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚127681062235136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59644444 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚121460676378624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151417101 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6406955395072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127849015438110 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚127681062235136 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151397035 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59644442 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚127681044232192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60536888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60536369 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226122752 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6518231424512 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135299114299392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BÇžÔO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 _runtime‚3407.633005902 YEDeviceStatsMonitor.oXç—&n_train_batch_start/allocated_bytes.all.allocated‚135492494673408 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 152990241 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚530 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153010281 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61161162 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143303766149632 _step‚374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129017261107200 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92745546 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35170904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6456421713350 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6475233566208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135492307670528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25990258 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143303531624960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6456586834754 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 $ _timestamp‚1.788410797220603e+09 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129188314026438 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25990786 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚122731892313088 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 65877068026885¾Eß'ŸÙO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚129188496187202 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60264733 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92725508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129017243104256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129017261107200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92725508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚135492494673408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61161692 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35170906 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60264733 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92745546 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6475064566272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚122731909352448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6475233566208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6475064566272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚135492307670528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6475233566208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135492307670528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25990258 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143303531624960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6456586834754 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 $ _timestamp‚1.788410797220603e+09 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129188314026438 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25990786 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚122731892313088 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6587706802688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 _runtime‚3407.633005902 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚135492494673408 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 152990241 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚530 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153010281 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61161162 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143303766149632 _step‚374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129017261107200 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92745546 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35170904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6456421713350 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 _runtime‚3408.295225669 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚129215646373318 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6457975999506 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6457810877894 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143333854940160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35178281 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25996159 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35178283 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92765359 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153022687 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61174442 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153022687 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚136744737770496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚135521157189632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4599 _step‚375 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6476457704448 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153042753 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60277394 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143334095742976 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6476626717696 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6476626717696 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚122757852534784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129044512468992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153042753 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6476457704448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61173915 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129215828534290 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚122757835495424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60277392 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60277394 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚135520970173440 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚136744761710592 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788410797882517e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129044530471936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129044530471936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92745295 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199†tÕ ¢ãM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143334095742976 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6476626717696 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6476626717696 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚122757852534784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129044512468992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153042753 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6476457704448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61173915 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129215828534290 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚122757835495424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60277392 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60277394 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚135520970173440 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚136744761710592 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 I@DeviceStatsMonitor.on_train_batch_end/allocatixÙ'mon.large_pool.peak‚2460 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788410797882517e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129044530471936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129044530471936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92745295 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92765359 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 _runtime‚3408.295225669 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚129215646373318 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6457975999506 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6457810877894 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143333854940160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35178281 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25996159 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35178283 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92765359 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153022687 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61174442 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153022687 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚136744737770496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚135521157189632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4599 _step‚375 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6476457704448 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153042753 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60277394 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0÷^òÖ®2£- _wandbruntime‚3408  _runtime‚3408G½ÒÎW¤Ñ .loss/FRt1/detection_loss‚3.048031768628529 /loss/FRt3/detection_loss‚1.8507609325647354 1loss/SRqqt1/assignment_loss‚2.704705343053148 1loss/FBt2/assignment_loss‚0.20505171269127231 1loss/SRqqt4/detection_loss‚1.8847780779555992 .loss/FBt2/detection_loss‚3.452912598848343 _runtime‚3408.296250039 /loss/FRt2/assignment_loss‚2.900987539026472 1loss/FBt1/assignment_loss‚0.47562764971662547 /loss/FRt2/detection_loss‚1.8740200641609372 1loss/SRqqt2/detection_loss‚1.3455722924125821 0loss/FRt3/assignment_loss‚2.7003054285049437 1loss/FBt3/assignment_loss‚0.19690532416594608 $loss/total_loss‚50.4898264651128 trainer/global_step‚4599 % _timestamp‚1.7884107978841534e+09 /loss/FRt4/assignment_loss‚4.109953682473365 2loss/SRqqt3/assignment_loss‚1.6344216506589542 2loss/SRqqt4/assignment_loss‚1.6048479183680482 0loss/FBt4/assignment_loss‚0.3990065875041182 0loss/SRqqt3/detection_loss‚1.851305109194734 /loss/FRt1/assignment_loss‚5.747680451188769 /loss/FBt1/detection_loss‚3.0621983647346496 _step‚376 .loss/FRt4/detection_loss‚1.960638481251739 1loss/SRqqt1/detection_loss‚2.4360300495817855 .loss/FBt3/detection_loss‚1.164207945216486 2loss/SRqqt2/assignment_loss‚1.2614348765315586 .loss/FBt4/detection_loss‚2.618442616679452 epoch‚3~ê|f\¥Ö 1loss/SRqqt1/assignment_loss‚2.704705343053148 1loss/FBt2/assignment_loss‚0.20505171269127231 1loss/SRqqt4/detection_loss‚1.8847780779555992 .loss/FBt2/detection_loss‚3.452912598848343 _runtime‚3408.296250039 /loss/FRt2/assignment_loss‚2.900987539026472 1loss/FBt1/assignment_loss‚0.47562764971662547 /loss/FRt2/detection_loss‚1.8740200641609372 1loss/SRqqt2/detection_loss‚1.3455722924125821 0loss/FRt3/assignment_loss‚2.7003054285049437 1loss/FBt3/assignment_loss‚0.19690532416594608 $loss/total_loss‚50.4898264651128 trainer/global_step‚4599 % _timestamp‚1.7884107978841534e+09 /loss/FRt4/assignment_loss‚4.109953682473365 2loss/SRqqt3/assignment_loss‚1.6344216506589542 2loss/SRqqt4/assignment_loss‚1.6048479183680482 0loss/FBt4/assignment_loss‚0.3990065875041182 0loss/SRqqt3/detection_loss‚1.851305109194734 /loss/FRt1/assignment_loss‚5.747680451188769 /loss/FBt1/detection_loss‚3.0621983647346496 _step‚376 .loss/FRt4/detection_loss‚1.960638481251739 1loss/SRqqt1/detection_loss‚2.4360300495817855 .loss/FBt3/detection_loss‚1.164207945216486 2loss/SRqqt2/assignment_loss‚1.2614348765315586 .loss/FBt4/detection_loss‚2.618442616679452 epoch‚3 .loss/FRt1/detection_loss‚3.048031768628529 /loss/FRt3/detection_loss‚1.8507609325647354øÆ‚H´œ¦‚(:‘ ¹÷ãÔ‘¤ùš gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.526' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.705199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m\4 §‚(:• ¹÷ãÔÈ÷œ disk.rbd1.in‚0 disk.rbd0.out‚0 network.sent‚ 1.0122672e+07% disk./.usageGB‚1097.5906028747559 disk.dm-0.in‚ 710.40625 disk.rbd0.in‚0 proc.cpu.threads‚17 network.recv‚ 1.264502e+06 disk.sda3.out‚ 144.50390625 disk.sda.out‚ 144.50390625 disk.sda1.out‚0# disk.dm-0.out‚57014.16748046875 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.rbd1.out‚ 1.90234375( disk.nvme0c0n1.out‚57014.16748046875 disk.sda3.in‚0 disk.nvme0n1.in‚ 710.40625 disk.drbd1777.out‚0 disk.rbd3.out‚0) disk./.usagePercent‚73.66022629878152 disk.dm-1.out‚ 144.50390625 disk.nvme0c0n1.in‚ 710.40625 disk.sda.in‚0$ memory_percent‚98.63109588623047% proc.memory.availableMB‚ 224.28125 disk.rbd2.out‚ 79.669921875 disk.sda2.out‚0* proc.memory.percent‚17.082595825195312 cpu‚25.33489535749277 disk.rbd2.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0 proc.memory.rssMB‚ 2798.8125 disk.rbd3.in‚0& disk.nvme0n1.out‚57014.16748046875ˆ3Ý–‘¨‚(:† È÷ãÔ ±Ï™ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.423' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0846 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«í&©‚(:› È÷ãÔ쨣œ disk.rbd0.out‚0( disk.nvme0c0n1.out‚57113.26513671875 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 710.40625* proc.memory.percent‚17.082595825195312 disk.sda2.out‚0 disk.sda3.out‚ 144.98046875# disk.dm-0.out‚57113.26513671875 disk.dm-1.out‚ 144.98046875 disk.sda3.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 710.40625 cpu‚25.333662595135795 proc.cpu.threads‚17 network.recv‚ 1.270572e+06 disk.rbd0.in‚0 disk.rbd2.out‚79.75634765625 disk.sda2.in‚0 disk.rbd1.out‚ 1.90234375 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 network.sent‚ 1.0230835e+07% disk./.usageGB‚1097.5906028747559 disk.rbd3.in‚0( proc.memory.availableMB‚ 224.49609375 proc.memory.rssMB‚ 2798.8125 disk.nvme0c0n1.in‚ 710.40625 disk.sda.out‚ 144.98046875& disk.nvme0n1.out‚57113.26513671875$ memory_percent‚98.62978458404541) disk./.usagePercent‚73.66022629878152 disk.dm-1.in‚03\S2ª- _wandbruntime‚3443  _runtime‚3443ê» ÕÇ«Á & lr-AdamW/pg2‚7.810020545451766e-05 trainer/global_step‚4649 % _timestamp‚1.7884108326071162e+09 _runtime‚3443.019271902 _step‚377 & lr-AdamW/pg1‚7.810020545451766e-05§¥ÅÌ¬Æ % _timestamp‚1.7884108326071162e+09 _runtime‚3443.019271902 _step‚377 & lr-AdamW/pg1‚7.810020545451766e-05 & lr-AdamW/pg2‚7.810020545451766e-05 trainer/global_step‚4649ùçò6I2­- _wandbruntime‚3443  _runtime‚3443êN#Ú'®ÔO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6526058653162 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93718210 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93718210 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35539756 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚378 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60897683 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35539754 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6525893531758 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93738248 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚130380711341056 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6658838147072 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154615893 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚144820099795968 trainer/global_step‚4649 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6544904888832 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154635933 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26261799 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26261279 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚138161496173568 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚130555127122410 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚124029051429888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚130380729344000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61801555 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6658627562496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6544735888896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61801033 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚136925447229952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚136925447229952 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚130554944961646 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚136925634232832 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6544904888832Nö;@a#¯ÙO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚130380711341056 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6658838147072 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154615893 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚144820099795968 trainer/global_step‚4649 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6544904888832 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154635933 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26261799 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26261279 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚138161496173568 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚130555127122410 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚124029051429888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚130380729344000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61801555 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6658627562496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6544735888896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61801033 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚136925447229952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚136925447229952 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚130554944961646 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚136925634232832 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6544904888832 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6526058653162 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93718210 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93718210 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35539756 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚378 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60897683 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35539754 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6525893531758 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93738248 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6546298040320 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26266793 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6527447817914 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138190385181696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚136954109732864 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6546129027072 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6660281403904 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3443.672142941 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154648339 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6660070832640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚136954296749056 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚130407998708736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 _step‚379 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚130407980705792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚130407980705792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60910342 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154648339 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 # _timestamp‚1.78841083325926e+09 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6546298040320 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154668405 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60910344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26267327 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚144850690525696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚124054994612224 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93758061 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61813924 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 trainer/global_step‚4649 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35547131 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003ƒdówè&²âM F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154668405 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚136954296749056 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚130407998708736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60910344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35547133 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚130582459469498 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6546298040320 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26266793 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6527447817914 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138190385181696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚136954109732864 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6546129027072 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6660281403904 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3443.672142941 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154648339 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6660070832640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚136954296749056 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚130407998708736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 _step‚379 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚130407980705792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚130407980705792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60910342 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154648339 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 # _timestamp‚1.78841083325926e+09 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6546298040320 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154668405 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60910344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26267327 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚144850690525696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚124054994612224 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93758061 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61813924 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 trainer/global_step‚4649 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35547131 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982ûl­\å2³- _wandbruntime‚3443  _runtime‚3443šû?¤Y´Ó % _timestamp‚1.7884108332614133e+09 2loss/SRqqt3/assignment_loss‚1.5456511726817832 0loss/FBt2/assignment_loss‚0.6567598588062538 1loss/FBt3/assignment_loss‚0.29894155497286884 /loss/FRt3/detection_loss‚1.9519259477570905 1loss/SRqqt2/detection_loss‚1.5226030384914742 .loss/FBt2/detection_loss‚3.212426396933469 .loss/FBt1/detection_loss‚2.793360847979784 _runtime‚3443.673460643 /loss/FRt2/detection_loss‚2.3507346853296807 .loss/FRt2/assignment_loss‚3.09138556110098 trainer/global_step‚4649 1loss/FBt4/assignment_loss‚0.48118955626809373 0loss/SRqqt1/detection_loss‚2.841554617440259 0loss/FRt3/assignment_loss‚2.2967701160301597 0loss/FRt4/assignment_loss‚3.3573378970225654 0loss/FBt1/assignment_loss‚0.4999548260548181 1loss/SRqqt3/detection_loss‚1.6574232056736946 .loss/FBt4/detection_loss‚2.703018360904285 epoch‚3 /loss/FRt1/assignment_loss‚4.731708830053156 2loss/SRqqt4/assignment_loss‚1.8747712119861886 /loss/FRt4/detection_loss‚2.2165543225904307 1loss/SRqqt4/detection_loss‚1.7731046969140019 2loss/SRqqt1/assignment_loss‚2.4935113301983587 2loss/SRqqt2/assignment_loss‚1.6789260929605614 .loss/FRt1/detection_loss‚2.930442154407501 /loss/FBt3/detection_loss‚0.8568298340050863 %loss/total_loss‚49.81688611656255 _step‚380иX^µØ 2loss/SRqqt1/assignment_loss‚2.4935113301983587 2loss/SRqqt2/assignment_loss‚1.6789260929605614 .loss/FRt1/detection_loss‚2.930442154407501 /loss/FBt3/detection_loss‚0.8568298340050863 %loss/total_loss‚49.81688611656255 _step‚380 % _timestamp‚1.7884108332614133e+09 2loss/SRqqt3/assignment_loss‚1.5456511726817832 0loss/FBt2/assignment_loss‚0.6567598588062538 1loss/FBt3/assignment_loss‚0.29894155497286884 /loss/FRt3/detection_loss‚1.9519259477570905 1loss/SRqqt2/detection_loss‚1.5226030384914742 .loss/FBt2/detection_loss‚3.212426396933469 .loss/FBt1/detection_loss‚2.793360847979784 _runtime‚3443.673460643 /loss/FRt2/detection_loss‚2.3507346853296807 .loss/FRt2/assignment_loss‚3.09138556110098 trainer/global_step‚4649 1loss/FBt4/assignment_loss‚0.48118955626809373 0loss/SRqqt1/detection_loss‚2.841554617440259 0loss/FRt3/assignment_loss‚2.2967701160301597 0loss/FRt4/assignment_loss‚3.3573378970225654 0loss/FBt1/assignment_loss‚0.4999548260548181 1loss/SRqqt3/detection_loss‚1.6574232056736946 .loss/FBt4/detection_loss‚2.703018360904285 epoch‚3 /loss/FRt1/assignment_loss‚4.731708830053156 2loss/SRqqt4/assignment_loss‚1.8747712119861886 /loss/FRt4/detection_loss‚2.2165543225904307 1loss/SRqqt4/detection_loss‚1.7731046969140019ü hm¶¶‚(:… ×÷ãÔüÍ™ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î­R½·‚(:ˆ ×÷ãÔÇ¢…œ disk.drbd1777.out‚0 disk.rbd1.in‚0$ memory_percent‚98.63147735595703 proc.memory.rssMB‚ 2798.8125 disk.sda2.out‚0 network.recv‚ 1.274288e+06% disk./.usageGB‚1097.5907249450684 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.sda.out‚ 145.59375 disk.sda1.in‚0 disk.sda1.out‚0( disk./.usagePercent‚73.6602344910226 disk.rbd2.out‚79.76806640625 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 710.40625% proc.memory.availableMB‚ 224.21875* proc.memory.percent‚17.082595825195312 proc.cpu.threads‚17 disk.dm-0.in‚ 710.40625 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 710.40625& disk.nvme0c0n1.out‚57113.806640625$ disk.nvme0n1.out‚57113.806640625 cpu‚25.332441556515906 disk.sda.in‚0 disk.rbd1.out‚ 1.90234375 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 145.59375 network.sent‚ 1.0236527e+07! disk.dm-0.out‚57113.806640625 disk.dm-1.out‚ 145.59375 disk.rbd0.out‚0¹uIœ¸‚(:‘ æ÷ãÔ¬¶™ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚120.138' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.027600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B&ôI$¹‚(:™ æ÷ãÔ«êÏ›& disk.nvme0n1.out‚57213.05517578125 disk.drbd1777.in‚0 proc.cpu.threads‚17# disk.dm-0.out‚57213.05517578125 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 145.98046875 disk.drbd1777.out‚0 cpu‚25.331233821110626 disk.sda1.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 224.4609375 network.recv‚ 1.281242e+06) disk./.usagePercent‚73.66025292356508 disk.dm-0.in‚ 710.40625 disk.dm-1.out‚ 145.98046875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚79.84619140625 network.sent‚ 1.0345649e+07 disk.nvme0n1.in‚ 710.40625 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 145.98046875 disk.rbd1.out‚ 1.90234375* proc.memory.percent‚17.082595825195312# memory_percent‚98.6299991607666% disk./.usageGB‚1097.5909996032715 disk.sda.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚57213.05517578125 disk.sda2.out‚0 disk.rbd1.in‚0 proc.memory.rssMB‚ 2798.8125±|p>2º- _wandbruntime‚3478  _runtime‚3478ÉÕÇ»Á trainer/global_step‚4699 % _timestamp‚1.7884108677253544e+09 _runtime‚3478.137488529 _step‚381 & lr-AdamW/pg1‚7.748637895579891e-05 & lr-AdamW/pg2‚7.748637895579891e-05œ÷ Ì¼Æ % _timestamp‚1.7884108677253544e+09 _runtime‚3478.137488529 _step‚381 & lr-AdamW/pg1‚7.748637895579891e-05 & lr-AdamW/pg2‚7.748637895579891e-05 trainer/global_step‚4699ýJ½$2½- _wandbruntime‚3478  _runtime‚3478˶=ÜÒ¾ÔO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94730950 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6729499688960 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚131744179577856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚131744197580800 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚131921758057618 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61530633 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6614407211520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62439789 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚138358773792256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚138358586789376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156261585 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35908604 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4699 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139607119644672 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6595530471570 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚514 ?9Deviceê#StatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204293120 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚138358773792256 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6614576211456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61530635 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _runtime‚3478.138526155 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚131744197580800 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131744179577856 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94710912 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139607143584768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156241545 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚138358586789376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62440305 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35908606 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚146336847566848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6614407211520 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6729703982080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61530635 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61530633 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6614576211456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚516 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94730950 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26531185 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6595365350166Öw–Þß'¿ÙO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61530633 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6614407211520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62439789 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚138358773792256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚138358586789376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156261585 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35908604 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4699 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139607119644672 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6595530471570 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚514 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204293120 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚138358773792256 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6614576211456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61530635 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _runtime‚3478.138526155 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚131744197580800 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131744179577856 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94710912 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139607143584768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156241545 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚138358586789376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62440305 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35908606 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚146336847566848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6614407211520 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6729703982080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61530635 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61530633 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6614576211456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚516 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94730950 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26531185 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6595365350166 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94730950 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6729499688960 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚131744179577856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚131744197580800 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚131921758057618 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360þ9P‹'2À- _wandbruntime‚3478  _runtime‚3478ÐŒ.ä&ÁÞM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61543292 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚138387249292288 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125352153729024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94750763 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125352170768384 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35915981 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94730699 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚138387436308480 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚131948908243734 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚131771448942592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6615969362944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6730913398272 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _step‚383 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚3478.797782492 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6731126066688 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156294057 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156273991 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 $ _timestamp‚1.788410868384976e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚131771466945536 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚131771466945536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156294057 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62452549 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6596754514710 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146367182599680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62453071 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61543292 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚138387436308480 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6615969362944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26537088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94750763 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6596919636322 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408id’gàÂãM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61543294 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 $ _timestamp‚1.788410868384976e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚131771466945536 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚131771466945536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156294057 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62452549 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6596754514710 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146367182599680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62453071 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61543292 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚138387436308480 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6615969362944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26537088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94750763 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6596919636322 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61543292 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚138387249292288 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125352153729024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94750763 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125352170768384 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35915981 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94730699 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚138387436308480 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚131948908243734 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 16901324¡ × 8 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚131771448942592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6615969362944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6730913398272 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _step‚383 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚3478.797782492 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6731126066688 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156294057 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156273991 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6684078534144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26802951 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚133288206832062 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36277456 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6665002289978 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6684247534080 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚133107647814656 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26802443 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚147853118989312 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚126623386702848 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62163583 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 157887237 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6664837168574 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚510 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141052767055872 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62163585 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141052790995968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36277454 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63079897 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚147853351416832 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚133288388992826 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62163585 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6684078534144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62163583 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6800351933440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95723652 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139791726348800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95703614 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6684247534080 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95703614 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884109030750148e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157867197 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚133107665817600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 trainer/global_step‚4749 _runtime‚3513.487316296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚139791913351680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139791913351680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚126623369663488 _step‚386 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63080407 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248âÅLà'ÑÚO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62163585 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141052790995968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36277454 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63079897 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚147853351416832 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚133288388992826 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62163585 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6684078534144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62163583 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6800351933440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95723652 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139791726348800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95703614 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6684247534080 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95703614 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884109030750148e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157867197 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚133107665817600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 trainer/global_step‚4749 _runtime‚3513.487316296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚139791913351680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139791913351680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚126623369663488 _step‚386 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63080407 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6684078534144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26802951 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚133288206832062 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36277456 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6665002289978 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6684247534080 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚133107647814656 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26802443 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚147853118989312 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚126623386702848 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62163583 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 157887237 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6664837168574 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚510 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141052767055872 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217‚T ø2Ò- _wandbruntime‚3514  _runtime‚3514:¨E¼ÓÞM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚147883444663808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62176244 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62176244 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6685471672320 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 157899643 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 PIc=j(DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3514.153683348 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚506 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95723401 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _step‚387 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚133134917179392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6685640685568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6685471672320 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26807714 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 trainer/global_step‚4749 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 157919709 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141081703944192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95743465 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6666391454730 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚139820575867904 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157919709 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26808220 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6801764659712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6801981522432 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36284831 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63092545 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6666226333118 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95723401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6685640685568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 157899643 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚139820388851712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚126649312845824 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 $ _timestamp‚1.788410903740957e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157919709 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26808220 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6801764659712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6801981522432 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36284831 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63092545 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6666226333118 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95723401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6685640685568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 157899643 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚139820388851712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚126649312845824 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 $ _timestamp‚1.788410903740957e+09 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62176244 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62176244 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6685471672320 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 157899643 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3514.153683348 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚506 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95723401 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _step‚387 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚133134917179392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6685640685568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6685471672320 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26807714 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 trainer/global_step‚4749 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 157919709 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141081703944192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95743465 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6666391454730 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2ƒ¯È’ 2Õ- _wandbruntime‚3514  _runtime‚3514Œž,,XÖÒ 0loss/FBt4/assignment_loss‚0.5489305620392163 1loss/SRqqt4/detection_loss‚1.9013910591602325 /loss/FBt3/detection_loss‚0.8615939467855395 _runtime‚3514.155262023 1loss/FBt3/assignment_loss‚0.24645943663908962 1loss/FBt2/assignment_loss‚0.34753054277971385 .loss/FRt4/detection_loss‚2.168650807166586 1loss/SRqqt2/detection_loss‚1.4323008733021247 epoch‚3 /loss/FBt4/detection_loss‚1.7000006516774495 0loss/FRt3/assignment_loss‚2.4759367138147352 0loss/FRt4/assignment_loss‚3.9558186689201666 1loss/SRqqt1/assignment_loss‚2.639785269132027 1loss/SRqqt4/assignment_loss‚1.993871275554685 %loss/total_loss‚51.22175626496405 $ _timestamp‚1.788410903743189e+09 1loss/SRqqt3/detection_loss‚1.5020001629988353 /loss/FBt2/detection_loss‚3.9039515018463136 trainer/global_step‚4749 .loss/FRt2/detection_loss‚2.261561196516542 2loss/SRqqt2/assignment_loss‚1.4229428943850704 0loss/FBt1/assignment_loss‚0.9167173794174398 1loss/SRqqt1/detection_loss‚3.1204764613738427 .loss/FBt1/detection_loss‚3.210603432221846 _step‚388 1loss/SRqqt3/assignment_loss‚1.625300163679101 /loss/FRt3/detection_loss‚2.0954693384468555 .loss/FRt1/detection_loss‚2.855009113297318 0loss/FRt2/assignment_loss‚2.9267561856438133 /loss/FRt1/assignment_loss‚5.108698628165505‘‰9Õ]×× %loss/total_loss‚51.22175626496405 $ _timestamp‚1.788410903743189e+09 1loss/SRqqt3/detection_loss‚1.5020001629988353 /loss/FBt2/detection_loss‚3.9039515018463136 trainer/global_step‚4749 .loss/FRt2/detection_loss‚2.261561196516542 2loss/SRqqt2/assignment_loss‚1.4229428943850704 0loss/FBt1/assignment_loss‚0.9167173794174398 1loss/SRqqt1/detection_loss‚3.1204764613738427 .loss/FBt1/detection_loss‚3.210603432221846 _step‚388 1loss/SRqqt3/assignment_loss‚1.625300163679101 /loss/FRt3/detection_loss‚2.0954693384468555 .loss/FRt1/detection_loss‚2.855009113297318 0loss/FRt2/assignment_loss‚2.9267561856438133 /loss/FRt1/assignment_loss‚5.108698628165505 0loss/FBt4/assignment_loss‚0.5489305620392163 1loss/SRqqt4/detection_loss‚1.9013910591602325 /loss/FBt3/detection_loss‚0.8615939467855395 _runtime‚3514.155262023 1loss/FBt3/assignment_loss‚0.24645943663908962 1loss/FBt2/assignment_loss‚0.34753054277971385 .loss/FRt4/detection_loss‚2.168650807166586 1loss/SRqqt2/detection_loss‚1.4323008733021247 epoch‚3 /loss/FBt4/detection_loss‚1.7000006516774495 0loss/FRt3/assignment_loss‚2.4759367138147352 0loss/FRt4/assignment_loss‚3.9558186689201666 1loss/SRqqt1/assignment_loss‚2.639785269132027 1loss/SRqqt4/assignment_loss‚1.993871275554685„ÖÀ”ßØ‚(:… ¢øãԞ駙 gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.248' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0496 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬Ü“ÊÙ‚(:“ ¢øãÔƧœ disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.in‚0 disk.rbd1.out‚ 1.90234375 disk.dm-0.in‚ 710.40625 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 147.95703125$ memory_percent‚98.63381385803223 proc.cpu.threads‚17 disk.sda.out‚ 147.95703125 disk.rbd1.in‚0' proc.memory.availableMB‚ 223.8359375* proc.memory.percent‚17.082595825195312 disk.sda3.out‚ 147.95703125 network.recv‚ 1.300188e+06 disk.rbd2.in‚0 disk.sda.in‚0 disk.drbd1777.out‚0) disk./.usagePercent‚73.66029362876303% disk./.usageGB‚1097.5916061401367 disk.rbd0.out‚0 proc.memory.rssMB‚ 2798.8125& disk.nvme0c0n1.out‚57511.994140625 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 710.40625! disk.dm-0.out‚57511.994140625 disk.rbd0.in‚0 network.sent‚ 1.0471641e+07 disk.rbd2.out‚ 81.6005859375 disk.drbd1777.in‚0$ disk.nvme0n1.out‚57511.994140625 cpu‚25.326540341550448‰º$NœÚ‚(:‘ ±øãÔ¦¡—™ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.467' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.293400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“ÊúéÛ‚(:” ±øãÔ®½ßœ disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 710.40625% disk./.usageGB‚1097.5916862487793 disk.sda1.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 710.40625 proc.cpu.threads‚17 network.sent‚ 1.0580563e+07 disk.dm-1.in‚0 disk.rbd0.in‚0 proc.memory.rssMB‚ 2798.8125# disk.dm-0.out‚57512.34716796875 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 224.015625 disk.rbd3.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚57512.34716796875 disk.sda3.out‚ 148.6171875 disk.sda.in‚0 disk.sda.out‚ 148.6171875 disk.sda3.in‚0* proc.memory.percent‚17.082595825195312 cpu‚25.32530025166825 network.recv‚ 1.306967e+06 disk.dm-1.out‚ 148.6171875 disk.rbd2.out‚ 81.6005859375& disk.nvme0n1.out‚57512.34716796875 disk.drbd1777.in‚0 disk.rbd1.out‚ 1.90234375 disk.nvme0n1.in‚ 710.40625 disk.sda2.in‚0$ memory_percent‚98.63271713256836) disk./.usagePercent‚73.66029900492124 disk.sda1.out‚0ˆù2Ü- _wandbruntime‚3548  _runtime‚3548–DÇÝÁ _step‚389 & lr-AdamW/pg1‚7.624199415798874e-05 & lr-AdamW/pg2‚7.624199415798874e-05 trainer/global_step‚4799 % _timestamp‚1.7884109380981205e+09 _runtime‚3548.510268977¡nsÌÞÆ _step‚389 & lr-AdamW/pg1‚7.624199415798874e-05 & lr-AdamW/pg2‚7.624199415798874e-05 trainer/global_step‚4799 % _timestamp‚1.7884109380981205e+09 _runtime‚3548.510268977…vÿ`B2ß- _wandbruntime‚3548  _runtime‚3548ßCïF"àÕO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 trainer/global_step‚4799 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159492849 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚142498438407168 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6871180224000 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6753749856768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96716354 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6753918856704 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62796533 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚141224865908224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚141225052911104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚127920528780288 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36646306 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96696316 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 _runtime‚3548.511266276 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚134471134054400 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚134471116051456 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6753918856704 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚134471134054400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36646304 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27073557 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159512889 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62796535 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62796533 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚149369833410048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚134471116051456 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚141224865908224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63719863 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6871395002880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6734474108386 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 C;DeviceStatsMonitor.Ûo8Þ•on_train_batch_start/segment.all.current‚416 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96716354 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149369594691072 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚134654837767270 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884109380988572e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27073023 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚134655019928034 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0挋ðà'áÚO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27073023 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚134655019928034 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 trainer/global_step‚4799 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159492849 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚142498438407168 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6871180224000 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6753749856768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96716354 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6753918856704 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62796533 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚141224865908224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚141225052911104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚127920528780288 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36646306 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96696316 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 _runtime‚3548.511266276 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚134471134054400 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚134471116051456 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6753918856704 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚134471134054400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36646304 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27073557 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159512889 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62796535 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62796533 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚149369833410048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚134471116051456 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚141224865908224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63719863 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6871395002880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6734474108386 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96716354 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149369594691072 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚134654837767270 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884109380988572e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536†âbý\2â- _wandbruntime‚3549  _runtime‚3549Åá7Öå&ãßM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _runtime‚3549.160480844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6755142994944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚127946489001984 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159545361 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159545361 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚141253715427328 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159525295 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚4799 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6755142994944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149399934604800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96716103 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚127946471962624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36653683 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6755312008192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6872807275008 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚141253715427328 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96736167 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96716103 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63732700 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63732162 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62809192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚134498385416192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27078481 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚142527327415296 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6872607189504 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6735698151526 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416eÍzðê&ääM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚4799 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6755142994944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149399934604800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96716103 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚127946471962624 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36653683 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6755312008192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6872807275008 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚141253715427328 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96736167 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96716103 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63732700 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63732162 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62809192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚134498385416192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27078481 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚142527327415296 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6872607189504 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6735698151526 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _runtime‚3549.160480844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6755142994944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚127946489001984 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159545361 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159545361 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚141253715427328 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159525295‡£—F2å- _wandbruntime‚3549  _runtime‚3549Éà°¡'æÒ 1loss/SRqqt1/detection_loss‚2.6078040037836345 /loss/FRt4/detection_loss‚2.4072489076190524 1loss/SRqqt4/detection_loss‚1.9887434989213943 % _timestamp‚1.7884109387495127e+09 2loss/SRqqt4/assignment_loss‚1.7872997211664916 0loss/FBt2/assignment_loss‚0.6530271946885478 /loss/FRt1/detection_loss‚2.9777815086500987 1loss/SRqqt2/detection_loss‚1.3027869086803459 epoch‚3 _step‚392 /loss/FBt3/detection_loss‚0.8142931576860039 /loss/FBt4/detection_loss‚2.3866468993100254 .loss/FRt3/detection_loss‚2.363103145506324 /loss/FRt1/assignment_loss‚4.826988875865936 /loss/FRt4/assignment_loss‚4.253172560532888 .loss/FRt2/detection_loss‚2.022026824299246 1loss/SRqqt3/detection_loss‚1.3450953486183332 /loss/FBt1/detection_loss‚3.2714524149894713 %loss/total_loss‚50.52301418900039 _runtime‚3549.161544018 1loss/FBt3/assignment_loss‚0.26486897634128875 0loss/FBt4/assignment_loss‚0.5748511010950262 trainer/global_step‚4799 0loss/FRt2/assignment_loss‚2.859160596504Ü›-18075 /loss/FRt3/assignment_loss‚2.505591828648637 1loss/SRqqt1/assignment_loss‚2.118998114551817 2loss/SRqqt2/assignment_loss‚1.5690624859729192 1loss/SRqqt3/assignment_loss‚1.360193516290377 0loss/FBt1/assignment_loss‚0.7720385822336538 .loss/FBt2/detection_loss‚3.490778017044067›pÛç]ç× trainer/global_step‚4799 0loss/FRt2/assignment_loss‚2.8591605965048075 /loss/FRt3/assignment_loss‚2.505591828648637 1loss/SRqqt1/assignment_loss‚2.118998114551817 2loss/SRqqt2/assignment_loss‚1.5690624859729192 1loss/SRqqt3/assignment_loss‚1.360193516290377 0loss/FBt1/assignment_loss‚0.7720385822336538 .loss/FBt2/detection_loss‚3.490778017044067 1loss/SRqqt1/detection_loss‚2.6078040037836345 /loss/FRt4/detection_loss‚2.4072489076190524 1loss/SRqqt4/detection_loss‚1.9887434989213943 % _timestamp‚1.7884109387495127e+09 2loss/SRqqt4/assignment_loss‚1.7872997211664916 0loss/FBt2/assignment_loss‚0.6530271946885478 /loss/FRt1/detection_loss‚2.9777815086500987 1loss/SRqqt2/detection_loss‚1.3027869086803459 epoch‚3 _step‚392 /loss/FBt3/detection_loss‚0.8142931576860039 /loss/FBt4/detection_loss‚2.3866468993100254 .loss/FRt3/detection_loss‚2.363103145506324 /loss/FRt1/assignment_loss‚4.826988875865936 /loss/FRt4/assignment_loss‚4.253172560532888 .loss/FRt2/detection_loss‚2.022026824299246 1loss/SRqqt3/detection_loss‚1.3450953486183332 /loss/FBt1/detection_loss‚3.2714524149894713 %loss/total_loss‚50.52301418900039 _runtime‚3549.161544018 1loss/FBt3/assignment_loss‚0.26486897634128875 0loss/FBt4/assignment_loss‚0.5748511010950262ˆê9m‘è‚(:† ÀøãÔó›Ö™ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.936' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1872 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç'"é‚(:’ ÀøãÔõÜÑœ disk.rbd0.in‚0( disk.nvme0c0n1.out‚57611.61669921875# disk.dm-0.out‚57611.61669921875 disk.rbd2.out‚ 83.3154296875 disk.rbd3.in‚0 disk.rbd3.out‚0$ memory_percent‚98.63426685333252 network.sent‚ 1.058587e+07 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 149.046875 disk.sda1.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.082595825195312 network.recv‚ 1.311111e+06$ disk./.usageGB‚1097.591667175293 disk.nvme0n1.in‚ 710.40625 disk.dm-0.in‚ 710.40625 disk.dm-1.in‚0 disk.sda2.out‚0 disk.drbd1777.in‚0 cpu‚25.324147958591016 disk.nvme0c0n1.in‚ 710.40625& disk.nvme0n1.out‚57611.61669921875 disk.rbd2.in‚0( proc.memory.availableMB‚ 223.76171875 proc.cpu.threads‚17) disk./.usagePercent‚73.66029772488358 disk.sda3.out‚ 149.046875 disk.drbd1777.out‚0 disk.rbd1.in‚0 proc.memory.rssMB‚ 2798.8125 disk.sda.out‚ 149.046875 disk.sda1.out‚0 disk.rbd1.out‚ 1.90234375óëÂÑê‚(:† ÏøãÔ¦À¶™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.687' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1374 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Äv9ø!ë‚(:– ÏøãԊ퉜 disk.sda.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 223.93359375) disk./.usagePercent‚73.66018508156851 disk.dm-0.in‚ 710.40625 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda3.in‚0* proc.memory.percent‚17.082595825195312 cpu‚25.323012249813573 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd2.out‚ 83.3154296875' disk.nvme0c0n1.out‚57711.0341796875 proc.memory.rssMB‚ 2798.8125 disk.sda1.in‚0" disk.dm-0.out‚57711.0341796875 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 149.69140625 network.recv‚ 1.317304e+06$ memory_percent‚98.63321781158447 network.sent‚ 1.0694333e+07$ disk./.usageGB‚1097.589988708496 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 710.40625 disk.dm-1.out‚ 149.69140625 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda.out‚ 149.69140625 disk.rbd1.in‚0 disk.rbd1.out‚ 1.90234375% disk.nvme0n1.out‚57711.0341796875 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 710.406257}„+ì‚(:„ ÞøãÔƒ„®™ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÁBåk í‚(:• ÞøãÔîÔ› disk.sda.out‚ 149.9921875 disk.sda1.out‚0 disk.nvme0n1.in‚ 710.40625 disk.rbd0.out‚0* proc.memory.percent‚17.571544647216797 disk.rbd3.out‚0 disk.rbd1.in‚0" disk.dm-0.out‚57711.4443359375 disk.dm-1.out‚ 149.9921875 disk.sda2.out‚0 disk.sda3.in‚0 cpu‚25.336636116022227 proc.cpu.threads‚17 disk.rbd1.out‚ 1.90234375( proc.memory.availableMB‚ 143.91796875 disk.rbd3.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 network.recv‚ 1.329332e+06 disk.sda1.in‚0% disk.nvme0n1.out‚57711.4443359375 network.sent‚ 1.0704273e+07 disk.rbd2.in‚0 disk.sda.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 710.40625$ memory_percent‚99.12159442901611 disk.rbd2.out‚ 83.3935546875 disk.nvme0c0n1.in‚ 710.40625' disk.nvme0c0n1.out‚57711.4443359375 disk.sda2.in‚0 disk.drbd1777.out‚0! proc.memory.rssMB‚ 2878.921875) disk./.usagePercent‚73.66018508156851$ disk./.usageGB‚1097.589988708496 disk.sda3.out‚ 149.9921875ü¼¤Sî‚(:… íøãԌ빙 gpu.0.gpu‚44.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€Ÿ>'ï‚(:œ íøãÔŽ¾•œ disk.nvme0n1.in‚ 710.40625* proc.memory.percent‚17.574167251586914 cpu‚25.356548097808748$ disk./.usageGB‚1097.590316772461 disk.rbd2.in‚0 disk.sda2.out‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.90234375" proc.memory.rssMB‚ 2879.3515625& disk.nvme0n1.out‚57810.89111328125 disk.rbd2.out‚ 83.3935546875 disk.sda3.in‚0 disk.sda3.out‚ 150.64453125 disk.sda.in‚0 disk.sda2.in‚0 network.recv‚ 1.332787e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 710.40625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 network.sent‚ 1.0710169e+07) disk./.usagePercent‚73.66020709821645 disk.drbd1777.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 150.64453125( disk.nvme0c0n1.out‚57810.89111328125 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda.out‚ 150.64453125$ memory_percent‚99.12469387054443( proc.memory.availableMB‚ 143.41015625 proc.cpu.threads‚17# disk.dm-0.out‚57810.89111328125 disk.rbd3.in‚0bØVð‚(:… üøãÔöÿ‹™ gpu.0.gpu‚36.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚97.827' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.5654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0鉟ñ‚(: üøãÔŠÀÚœ disk.rbd0.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚ 710.40625 disk.dm-0.in‚ 710.40625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.drbd1777.in‚0 disk.drbd1777.out‚0" proc.memory.rssMB‚ 2879.3515625% disk./.usageGB‚1097.5904655456543( disk.nvme0c0n1.out‚57811.26611328125& disk.nvme0n1.out‚57811.26611328125 cpu‚25.37416636747274) disk./.usagePercent‚73.66021708251029 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.90234375# memory_percent‚99.1140365600586% proc.memory.availableMB‚ 145.15625* proc.memory.percent‚17.574167251586914 network.sent‚ 1.0714776e+07 network.recv‚ 1.336775e+06 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda3.out‚150.9375 proc.cpu.threads‚17# disk.dm-0.out‚57811.26611328125 disk.dm-1.out‚150.9375 disk.rbd2.in‚0 disk.rbd2.out‚83.49560546875 disk.sda2.out‚0 disk.sda.out‚150.9375 disk.sda1.out‚0FQÌ2ò- _wandbruntime‚3621  _runtime‚3621,Õ 3A*ó»T LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2460 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 269877760 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚136560548952206 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚143226944359936 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 161601759 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚416 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 161585780 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚64751148 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚217 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 6940913604608 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚129757619695616 trainer/global_step‚4835 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 161601759 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚151469681868800 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _step‚393 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 $ _timestamp‚1.788411010732513e+09 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚63789806 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 145311232 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚37237755 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚982 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 406789969 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 417333248 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 6822595017728 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚217 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚199 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚136404476650496 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2460 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚27513395 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 4757245257 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 4390226944 _runtime‚3621.145317884 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚64751434 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 4912020992 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚136404494653440 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 6941029565952 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 5041553408 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚136404476650496 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 140964952 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚199 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚143227089671168 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 123925592 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 417333248 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚97795976 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1003 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 6802805330998 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 4624220160 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚63789804 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚27513679 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚217 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 127308288 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 127308288 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚97811953 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚416 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚286 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 6822467709440 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 410712064 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚199 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚129757619695616 trainer/global_step‚4835 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 161601759 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚151469681868800 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _step‚393 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 $ _timestamp‚1.788411010732513e+09 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚63789806 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 145311232 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚37237755 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚982 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 406789969 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 417333248 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 6822595017728 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚217 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚199 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚136404476650496 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2460 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚27513395 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 4757245257 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 4390226944 _runtime‚3621.145317884 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚64751434 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 4912020992 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚136404494653440 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 6941029565952 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 5041553408 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚136404476650496 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 140964952 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚199 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416‰ÍŒÑà2õ- _wandbruntime‚3622  _runtime‚3622“_²°¢öÅR G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 127308288 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚97803372 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚27518663 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚199 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 410712064 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚64762632 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚97803372 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚143246158681600 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚136423359839232 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚136423341836288 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 6941262740480 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚199 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚143246158681600 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚416 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 4912020992 WJDeviceStatsMonitor.on_validation_batcÆuH]©!h_end/allocation.small_pool.allocated‚97819349 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚286 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 trainer/global_step‚4835 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 6822944153600 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚37244255 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 6822944153600 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚37244253 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 _step‚394 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚129775521249280 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 161603888 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 139901440 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚144548395769856 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 4757245257 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚416 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚416 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 145311232 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚63800516 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 123925592 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 6803152937334 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 5041553408 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 4624220160 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚64762918 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 6941378701824 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚143246303992832 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 417333248 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚63800516 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 4912020992 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚136578798112206 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 5041553408 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚143246303992832 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚97819349 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚136423359839232 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 4541568000 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 269877760 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 $ _timestamp‚1.788411011634745e+09 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚27518379 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚982 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚217 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 161619867 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 140964952 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 161619867 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2460 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 6803276862926 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚63800518 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 145311232 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚129775504209920 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 5041553408 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚151489634570240 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 127308288 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 410712064 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1003 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚144548371829760 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 161603888 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚151489774471680 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚64762918 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 6941378701824 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚143246303992832 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 417333248 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚63800516 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 4912020992 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚136578798112206 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 5041553408 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚143246303992832 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚97819349 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚136423359839232 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 4541568000 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 269877760 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 $ _timestamp‚1.788411011634745e+09 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚27518379 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚982 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚217 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 161619867 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 140964952 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 161619867 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2460 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 6803276862926 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚63800518 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 145311232 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚129775504209920 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 5041553408 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚151489634570240 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 127308288 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 410712064 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1003 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚144548371829760 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 161603888 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚151489774471680 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 B(ù‚(: ‹ùãÔÑõºœ disk.rbd2.out‚83.49560546875* proc.memory.percent‚17.574167251586914 disk.sda.out‚ 151.57421875 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 710.40625 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.rbd3.in‚0# disk.dm-0.out‚57911.43017578125" proc.memory.rssMB‚ 2879.3515625 cpu‚25.39438743510412 proc.cpu.threads‚17 network.recv‚ 1.340709e+06 disk.sda.in‚0 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.out‚0 disk.sda3.out‚ 151.57421875$ memory_percent‚99.11630153656006 network.sent‚ 1.0720635e+07% disk./.usageGB‚1097.5906600952148& disk.nvme0n1.out‚57911.43017578125 disk.sda3.in‚0) disk./.usagePercent‚73.66023013889453 disk.dm-1.out‚ 151.57421875 disk.sda2.in‚0( proc.memory.availableMB‚ 144.78515625 disk.sda1.out‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚57911.43017578125 disk.sda1.in‚0 disk.rbd1.out‚ 1.90234375 disk.dm-0.in‚ 710.40625 disk.rbd0.in‚0 disk.rbd0.out‚0ªÝ°2ú- _wandbruntime‚3634  _runtime‚3634\0ò.û¥ :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.40314310789108276 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 )jet/accuracy_3_of_4‚0.048492182046175 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3981344997882843 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.31172841787338257 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 +particle/specificity‚0.9475602507591248 jet/accuracy_1_of_7‚1 trainer/global_step‚4835 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9739693999290466 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3204615116119385 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4573412537574768 particle/accuracy_1_of_5‚1 jet/accuracy_4_of_6‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9921246767044067 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3159843981266022 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.38375234603881836 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.39149293303489685 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.3879629671573639 particle/accuracy_4_of_4‚1 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1133333295583725 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 )jet/accuracy_1_of_1‚0.421040803194046 particle/accuracy_3_of_7‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22454550862312317 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5113301873207092 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 particle/accuracy_5_of_5‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9896560907363892 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12796099483966827 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17551477253437042 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6727448105812073 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9869115352630615 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.19747474789619446 particle/accuracy_3_of_3‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ) Purity/4FRt0SRqqt*FBt/FRt_purity‚0.25 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9906432628631592 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.126460462808609 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9599999785423279 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.002809359924867749 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.47174081206321716 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20383727550506592 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.007741817273199558 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6452004909515381 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1944444328546524 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.30033546686172485 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.28919535875320435 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4728233814239502 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1619059145450592 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.35185185074806213 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.10700049251317978 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.025755248963832855 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1761261373758316 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,jet/accuracy_4_of_4‚0.004491487983614206 particle/accuracy_3_of_5‚1 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5350877642631531 ) Purity/4FRt0SRqqt0FBt/FRt_purity‚0.25 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9963440299034119 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9920569658279419 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.35596343874931335 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.19942528009414673 *jet/accuracy_1_of_4‚0.8538680076599121 *jet/accuracy_2_of_4‚0.2986763119697571 *jet/accuracy_2_of_5‚0.4159626066684723 ,jet/accuracy_4_of_5‚0.006720430217683315 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6515151262283325 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03988095372915268 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9941622614860535 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012321513146162033 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3866666853427887 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 particle/accuracy_4_of_7‚1 7"Purity/0FRt0SRqqt2FBt/event_purity‚0.39382404088974 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6515151262283325 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07649781554937363 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.040566038340330124 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1805555671453476 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ,jet/accuracy_3_of_6‚0.057692307978868484 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.45577272772789 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02500000037252903 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.125 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4399999976158142 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1480434089899063 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998066782951355 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.13601532578468323 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 jet/accuracy_5_of_7‚0 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ("Purity/1FRt1SRqqt3FBt/event_purit:ߤ¥cy‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.21171928942203522 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.04363124072551727 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.28925755620002747 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.42505940794944763 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1111111119389534 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.36370131373405457 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19764195382595062 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9597222805023193 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15572775900363922 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4952380657196045 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 jet/accuracy_7_of_7‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.33339619636535645 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.37068966031074524 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9866275191307068 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 particle/accuracy_4_of_6‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14353585243225098 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5658866763114929 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 particle/accuracy_5_of_7‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996220469474792 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.37914156913757324 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.13636364042758942 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3603895902633667 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 particle/accuracy_1_of_4‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5259600877761841 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9797636270523071 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 (particle/f_score‚0.37605664134025574 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4791666567325592 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.019152337685227394 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.35407498478889465 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19969865679740906 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 _runtime‚3634.737793317 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10638298094272614 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.39324766397476196 particle/accuracy_3_of_4‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.0625 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 particle/accuracy_7_of_7‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.484725683927536 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2278001457452774 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.018820861354470253 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.31060606241226196 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.050287358462810516 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 *jet/accuracy_1_of_6‚0.9615384340286255 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.37914156913757324 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21783946454524994 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9802111983299255 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9572964310646057 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.14016173779964447 *jet/accuracy_1_of_3‚0.8102466464042664 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21263036131858826 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.33591949939727783 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04288171976804733 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_7‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3241482079029083 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_3‚0.2145431488752365 +jet/accuracy_2_of_6‚0.37179484963417053 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3292617201805115 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.537544846534729 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9468390345573425 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006621801294386387 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 _step‚395 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.995573878288269 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19782604277133942 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19994451105594635 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.42151322960853577 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.44540807604789734 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0350070558488369 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9891883134841919 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.019217688590288162 8"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.321159690618515 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5533248782157898 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14558768272399902 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2729315757751465 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.16754385828971863 7"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.36257404088974 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4999999701976776 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.43543514609336853 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_1‚1 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.44236111640930176 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666865348816 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9727854132652283 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.36635804176330566 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_3‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 5 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.99017333984375 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3935646116733551 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.37938597798347473 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.019495565444231033 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 +jet/accuracy_3_of_5‚0.13200846314430237 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3935185372829437 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9932476878166199 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4129391312599182 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.126460462808609 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1619001179933548 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.1997889280319214 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.37209659814834595 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.36983323097229004 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.30751630663871765 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.09375 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8261991739273071 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2061586081981659 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3361110985279083 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1511223167181015 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 % _timestamp‚1.7884110243236115e+09 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_4‚1 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8859648704528809 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.28602656722068787 *jet/accuracy_1_of_2‚0.6925998330116272 -"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1875 particle/accuracy_6_of_7‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.39646589756011963 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9125296473503113 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1735442876815796 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1527077704668045 ,particle/sensitivity‚0.29742351174354553 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8333333134651184 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.48587900400161743 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3641975224018097 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.366818368434906 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1545977145433426 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 jet/accuracy_6_of_7‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1512458175420761 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16661259531974792 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994527101516724 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 6 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.364806592464447 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12280171364545822 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_3‚1 particle/accuracy_2_of_6‚1 jet/accuracy_4_of_7‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5157805681228638 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.18715238571166992 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3930129110813141 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10539407283067703 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3642117381095886 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.13835468888282776 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.347222238779068 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 particle/accuracy_3_of_6‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4999999701976776 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.3095132112503052 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.130952388048172 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.31755951046943665 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_2_of_7‚0.5 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.17184236645698547 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.017960214987397194 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.17914390563964844 *jet/accuracy_1_of_5‚0.9115207195281982 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1356411725282669 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994527101516724 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.761904776096344 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_2‚0.1024903804063797 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15306492149829865 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9833835959434509 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3723599314689636 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3306306004524231 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06578947603702545 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01766485720872879 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.41925284266471863 particle/accuracy_5_of_6‚1 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.0625 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05903920531272888 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3439922332763672 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3204491436481476 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2750000059604645 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.46845242381095886 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9848607778549194 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_2_of_2‚1 'particle/accuracy‚0.846049427986145 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3009259104728699 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1518694907426834 ,jet/accuracy_3_of_3‚0.021323898807168007 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3571428656578064 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19362035393714905 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3675083518028259 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3021157383918762 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3568948209285736 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2240740805864334 epoch‚3 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4909946024417877 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5333333611488342 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7validation_average_jet_accuracy‚0.39055415987968445 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 (loss/val_total_loss‚50.6838459099427 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40225139260292053 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15720385313034058 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.44811829924583435 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9934523701667786 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.32777777314186096 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3058822751045227 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.43396657705307007 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2013888955116272 particle/accuracy_2_of_7‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14403897523880005 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3724209666252136 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07616250962018967 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.12962962687015533 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9877218008041382 5 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.98005211353302 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1205243319272995 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1111111119389534 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.41276293992996216 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1!‰dMü•¥ :"Purity/*FRt0SRqqt2FBt/event_purity‚0.28602656722068787 *jet/accuracy_1_of_2‚0.6925998330116272 -"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1875 particle/accuracy_6_of_7‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.39646589756011963 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9125296473503113 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1735442876815796 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1527077704668045 ,particle/sensitivity‚0.29742351174354553 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8333333134651184 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.48587900400161743 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3641975224018097 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.366818368434906 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1545977145433426 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 jet/accuracy_6_of_7‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1512458175420761 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16661259531974792 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994527101516724 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 6 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.364806592464447 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12280171364545822 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_3‚1 particle/accuracy_2_of_6‚1 jet/accuracy_4_of_7‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5157805681228638 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.18715238571166992 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3930129110813141 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10539407283067703 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3642117381095886 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.13835468888282776 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.347222238779068 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 particle/accuracy_3_of_6‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4999999701976776 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.3095132112503052 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.130952388048172 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.31755951046943665 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_2_of_7‚0.5 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.17184236645698547 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.017960214987397194 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.17914390563964844 *jet/accuracy_1_of_5‚0.9115207195281982 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1356411725282669 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994527101516724 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.761904776096344 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_2‚0.1024903804063797 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15306492149829865 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9833835959434509 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3723599314689636 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3306306004524231 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06578947603702545 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01766485720872879 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.41925284266471863 particle/accuracy_5_of_6‚1 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.0625 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05903920531272888 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3439922332763672 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3204491436481476 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2750000059604645 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.46845242381095886 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9848607778549194 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_2_of_2‚1 'particle/accuracy‚0.846049427986145 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3009259104728699 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1518694907426834 ,jet/accuracy_3_of_3‚0.021323898807168007 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3571428656578064 ("Purity/2FRt3SRqqt1FBt/ev›V‘ÀOvent_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19362035393714905 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3675083518028259 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3021157383918762 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3568948209285736 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2240740805864334 epoch‚3 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4909946024417877 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5333333611488342 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7validation_average_jet_accuracy‚0.39055415987968445 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 (loss/val_total_loss‚50.6838459099427 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40225139260292053 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15720385313034058 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.44811829924583435 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9934523701667786 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.32777777314186096 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3058822751045227 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.43396657705307007 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2013888955116272 particle/accuracy_2_of_7‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14403897523880005 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3724209666252136 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07616250962018967 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.12962962687015533 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9877218008041382 5 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.98005211353302 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1205243319272995 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1111111119389534 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.41276293992996216 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.40314310789108276 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 )jet/accuracy_3_of_4‚0.048492182046175 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3981344997882843 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.31172841787338257 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 +particle/specificity‚0.9475602507591248 jet/accuracy_1_of_7‚1 trainer/global_step‚4835 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9739693999290466 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3204615116119385 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4573412537574768 particle/accuracy_1_of_5‚1 jet/accuracy_4_of_6‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9921246767044067 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3159843981266022 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.38375234603881836 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.39149293303489685 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.3879629671573639 particle/accuracy_4_of_4‚1 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1133333295583725 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 )jet/accuracy_1_of_1‚0.421040803194046 particle/accuracy_3_of_7‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22454550862312317 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5113301873207092 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 particle/accuracy_5_of_5‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9896560907363892 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12796099483966827 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17551477253437042 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6727448105812073 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9869115352630615 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.19747474789619446 particle/accuracy_3_of_3‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ) Purity/4FRt0SRqqt*FBt/FRt_purity‚0.25 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9906432628631592 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.126460462808609 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9599999785423279 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.002809359924867749 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.47174081206321716 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20383727550506592 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.007741817273199558 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6452004909515381 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1944444328546524 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.30033546686172485 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.28919535875320435 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4728233814239502 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1619059145450592 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.35185185074806213 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.10700049251317978 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.025755248963832855 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1761261373758316 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,jet/accuracy_4_of_4‚0.004491487983614206 particle/accuracy_3_of_5‚1 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5350877642631531 ) Purity/4FRt0SRqqt0FBt/FRt_purity‚0.25 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9963440299034119 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9920569658279419 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.35596343874931335 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.19942528009414673 *jet/accuracy_1_of_4‚0.8538680076599121 *jet/accuracy_2_of_4‚0.2986763119697571 *jet/accuracy_2_of_5‚0.4159626066684723 ,jet/accuracy_4_of_5‚0.006720430217683315 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6515151262283325 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03988095372915268 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9941622614860535 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012321513146162033 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3866666853427887 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 particle/accuracy_4_of_7‚1 7"Purity/0FRt0SRqqt2FBt/event_purity‚0.39382404088974 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6515151262283325 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07649781554937363 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.040566038340330124 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1805555671453476 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ,jet/accuracy_3_of_6‚0.057692307978868484 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.45577272772789 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02500000037252903 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.125 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4399999976158142 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1480434089899063 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998066782951355 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.13601532578468323 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 jet/accuracy_5_of_7‚0 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.21171928942203522 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.04363124072551727 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.28925755620002747 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.42505940794944763 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1111111119389534 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.36370131373405457 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19764195382595062 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9597222805023193 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15572775900363922 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4952380657196045 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 jet/accuracy_7_of_7‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.33339619636535645 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.37068966031074524 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9866275191307068 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 particle/accuracy_4_of_6‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14353585243225098 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5658866763114929 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 particle/accuracy_5_of_7‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996220469474792 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.37914156913757324 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.13636364042758942 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3603895902633667 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 particle/accuracy_1_of_4‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5259600877761841 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9797636270523071 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 (particle/f_score‚0.37605664134025574 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4791666567325592 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.019152337685227394 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.35407498478889465 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19969865679740906 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 _runtime‚3634.737793317 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10638298094272614 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.39324766397476196 particle/accuracy_3_of_4‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.0625 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 particle/accuracy_7_of_7‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.484725683927536 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2278001457452774 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.018820861354470253 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.31060606241226196 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.050287358462810516 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 *jet/accuracy_1_of_6‚0.9615384340286255 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.37914156913757324 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21783946454524994 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9802111983299255 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9572964310646057 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.14016173779964447 *jet/accuracy_1_of_3‚0.8102466464042664 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21263036131858826 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.33591949939727783 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04288171976804733 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_7‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3241482079029083 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_3‚0.2145431488752365 +jet/accuracy_2_of_6‚0.37179484963417053 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3292617201805115 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.537544846534729 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9468390345573425 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006621801294386387 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 _step‚395 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.995573878288269 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19782604277133942 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19994451105594635 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.42151322960853577 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.44540807604789734 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0350070558488369 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9891883134841919 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.019217688590288162 8"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.321159690618515 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5533248782157898 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14558768272399902 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2729315757751465 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.16754385828971863 7"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.36257404088974 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4999999701976776 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.43543514609336853 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_1‚1 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.44236111640930176 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666865348816 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9727854132652283 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.36635804176330566 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_3‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 5 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.99017333984375 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3935646116733551 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.37938597798347473 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.019495565444231033 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 +jet/accuracy_3_of_5‚0.13200846314430237 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3935185372829437 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9932476878166199 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4129391312599182 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.126460462808609 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1619001179933548 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.1997889280319214 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.37209659814834595 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.36983323097229004 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.30751630663871765 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.09375 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8261991739273071 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2061586081981659 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3361110985279083 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1511223167181015 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 % _timestamp‚1.7884110243236115e+09 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_4‚1 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8859648704528809‹=aý‚(:… šùãÔ·Ž™ gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(|þ‚(: šùãÔã厜 network.recv‚ 1.355482e+06 disk.sda2.out‚0 disk.sda.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2879.3515625 cpu‚25.40006977167316) disk./.usagePercent‚73.66026367588152 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda3.out‚ 152.05078125 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 1.90234375 network.sent‚ 1.0833153e+07 disk.sda2.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 710.40625 proc.cpu.threads‚17 disk.sda3.in‚0" disk.nvme0n1.out‚ 58133.6484375 disk.rbd2.in‚0 disk.rbd3.out‚0$ disk.nvme0c0n1.out‚ 58133.6484375 disk.sda.out‚ 152.05078125 disk.sda1.out‚0 disk.dm-0.out‚ 58133.6484375 disk.dm-1.in‚0$ memory_percent‚97.71904945373535 disk.dm-0.in‚ 710.40625 disk.rbd0.in‚0' proc.memory.availableMB‚ 373.7109375% disk./.usageGB‚1097.5911598205566 disk.dm-1.out‚ 152.05078125 disk.rbd2.out‚85.21044921875* proc.memory.percent‚17.574167251586914 disk.drbd1777.in‚0´4m2ÿ- _wandbruntime‚3646  _runtime‚3646±_û®Ç€Á & lr-AdamW/pg1‚7.561169518682407e-05 & lr-AdamW/pg2‚7.561169518682407e-05 trainer/global_step‚4849 % _timestamp‚1.7884110359847972e+09 _runtime‚3646.396980935 _step‚396 ¥›ŒÌÆ _step‚396 & lr-AdamW/pg1‚7.561169518682407e-05 & lr-AdamW/pg2‚7.561169518682407e-05 trainer/global_step‚4849 % _timestamp‚1.7884110359847972e+09 _runtime‚3646.396980935Œ50S2‚- _wandbruntime‚3646  _runtime‚3646*ûjØ׃ÕO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137023108996096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚143868711726080 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚137171182889856 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitorñ¡Ü¡&.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚137171365050620 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4849 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162281282 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64104341 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚143868524723200 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152143312882176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6964226373120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚505 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚145179086509056 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162261242 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37424656 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 % _timestamp‚1.7884110359855833e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98156903 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137023090993152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65074856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6845433730048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6845602729984 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98176941 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27650707 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98156903 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6964443249152 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚397 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 _runtime‚3646.397963998 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64104339 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37424654 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6845602729984 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65075363 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152143553698304 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚145179110449152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 Âà'„ÚO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚143868524723200 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152143312882176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6964226373120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚505 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚145179086509056 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162261242 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37424656 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 % _timestamp‚1.7884110359855833e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98156903 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137023090993152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65074856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6845433730048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6845602729984 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98176941 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27650707 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98156903 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6964443249152 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚397 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 _runtime‚3646.397963998 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64104339 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37424654 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6845602729984 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65075363 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152143553698304 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚145179110449152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137023108996096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚143868711726080 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚137171182889856 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚137171365050620 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4849 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162281282 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64104341 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4849 _step‚398 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162313754 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6846995881472 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚145207999457280 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚130371428960256 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6827251398092 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚137050378360832 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162293688 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6846826868224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6965640043520 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64117000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65087404 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27655373 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37432033 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162293688 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152173639500800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64117000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚143897187226112 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚145208023397376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 % _timestamp‚1.7884110366313748e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚143897374242304 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64116998 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98196754 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚137198697397708 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6827086276480 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚143897374242304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98196754 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37432031 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚137198515236736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199È¡Uâ ‡äM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64116998 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98196754 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚137198697397708 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6827086276480 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚143897374242304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98196754 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37432031 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚137198515236736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6846995881472 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98176690 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27655862 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4849 _step‚398 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162313754 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6846995881472 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 _KDeviceStatsMonitor.on_train_ba«×[Ntch_end/inactive_split_bytes.large_pool.freed‚145207999457280 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚130371428960256 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6827251398092 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚137050378360832 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162293688 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6846826868224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6965640043520 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64117000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65087404 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27655373 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37432033 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162293688 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152173639500800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64117000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚143897187226112 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚145208023397376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 % _timestamp‚1.7884110366313748e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚143897374242304Ž² v%2ˆ- _wandbruntime‚3647  _runtime‚36478¨([‰Õ /loss/FBt2/detection_loss‚3.4469621658325194 0loss/FRt2/assignment_loss‚3.9515079886485367 2loss/SRqqt2/assignment_loss‚1.4566815927973948 1loss/FBt4/assignment_loss‚0.33744563703881936 /loss/FRt4/detection_loss‚1.9511967645751105 0loss/SRqqt3/detection_loss‚1.328606702522798 epoch‚4 % _timestamp‚1.7884110366336443e+09 1loss/SRqqt2/detection_loss‚1.5827461427543312 /loss/FBt3/detection_loss‚0.9965367119891906 /loss/FBt4/detection_loss‚2.6986446334765506 1loss/FBt1/assignment_loss‚0.30427559709187335 trainer/global_step‚4849 2loss/SRqqt3/assignment_loss‚1.5004433988000858 .loss/FRt3/detection_loss‚1.880529755895788 %loss/total_loss‚49.28601668149981 /loss/FRt4/assignment_loss‚3.858369161023034 0loss/FBt2/assignment_loss‚0.3956647180930304 .loss/FRt2/detection_loss‚2.548443172222529 1loss/SRqqt1/detection_loss‚2.6505280711032726 .loss/FBt1/detection_loss‚3.348337948322296 2loss/SRqqt4/assignment_loss‚1.4463273113194322 _runtime‚ 3647.04573105 _step‚399 0loss/FRt3/assignment_loss‚2.3664490553465756 2loss/SRqqt1/assignment_loss‚2.1467134621408253 /loss/FRt1/assignment_loss‚4.279510217554429 1loss/FBt3/assignment_loss‚0.31048160354263943 /loss/FRt1/detection_loss‚2.8004234257866356 1loss/SRqqt4/detection_loss‚1.6991914436221123k Š`ŠÚ 2loss/SRqqt1/assignment_loss‚2.1467134621408253 /loss/FRt1/assignment_loss‚4.279510217554429 1loss/FBt3/assignment_loss‚0.31048160354263943 /loss/FRt1/detection_loss‚2.8004234257866356 1loss/SRqqt4/detection_loss‚1.6991914436221123 /loss/FBt2/detection_loss‚3.4469621658325194 0loss/FRt2/assignment_loss‚3.9515079886485367 2loss/SRqqt2/assignment_loss‚1.4566815927973948 1loss/FBt4/assignment_loss‚0.33744563703881936 /loss/FRt4/detection_loss‚1.9511967645751105 0loss/SRqqt3/detection_loss‚1.328606702522798 epoch‚4 % _timestamp‚1.7884110366336443e+09 1loss/SRqqt2/detection_loss‚1.5827461427543312 /loss/FBt3/detection_loss‚0.9965367119891906 /loss/FBt4/detection_loss‚2.6986446334765506 1loss/FBt1/assignment_loss‚0.30427559709187335 trainer/global_step‚4849 2loss/SRqqt3/assignment_loss‚1.5004433988000858 .loss/FRt3/detection_loss‚1.880529755895788 %loss/total_loss‚49.28601668149981 /loss/FRt4/assignment_loss‚3.858369161023034 0loss/FBt2/assignment_loss‚0.3956647180930304 .loss/FRt2/detection_loss‚2.548443172222529 1loss/SRqqt1/detection_loss‚2.6505280711032726 .loss/FBt1/detection_loss‚3.348337948322296 2loss/SRqqt4/assignment_loss‚1.4463273113194322 _runtime‚ 3647.04573105 _step‚399 0loss/FRt3/assignment_loss‚2.3664490553465756T«”N‹‚(:… ©ùãԤ燙 gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚97.578' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.5156 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú.͘&Œ‚(:› ©ùãÔâͤœ$ memory_percent‚97.71356582641602& proc.memory.availableMB‚ 374.609375 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 710.40625 disk.dm-1.out‚ 152.72265625 disk.rbd3.in‚0 network.sent‚ 1.0957216e+07 disk.rbd2.in‚0 disk.rbd2.out‚85.21044921875 disk.nvme0c0n1.in‚ 710.40625 disk.sda1.in‚0* proc.memory.percent‚17.568159103393555 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 152.72265625 disk.dm-0.in‚ 710.40625# disk.dm-0.out‚58257.49365234375 disk.rbd0.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚58257.49365234375 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.90234375& disk.nvme0n1.out‚58257.49365234375" proc.memory.rssMB‚ 2878.3671875 proc.cpu.threads‚17) disk./.usagePercent‚73.66023039490207 disk.sda2.out‚0 disk.sda.in‚0 disk.rbd0.out‚0 cpu‚25.398642351154773 network.recv‚ 1.362026e+06$ disk./.usageGB‚1097.590663909912 disk.sda.out‚ 152.72265625 disk.dm-1.in‚0ÛÌÛ‘‚(:† ¸ùãԻЧ™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.687' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1374 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ5ë4Ž‚(:” ¸ùãÔº—ˆœ network.recv‚ 1.367557e+06 disk.sda1.out‚0 disk.rbd1.out‚ 1.98828125& disk.nvme0n1.out‚58356.56005859375 disk.rbd3.out‚0# disk.dm-0.out‚58356.56005859375 disk.rbd2.in‚0 disk.sda3.out‚ 153.32421875 disk.sda.out‚ 153.32421875" proc.memory.rssMB‚ 2878.3671875* proc.memory.percent‚17.568159103393555) disk./.usagePercent‚73.66023039490207 disk.rbd1.in‚0 disk.rbd3.in‚0 cpu‚25.39723177381779 disk.drbd1777.out‚0 disk.dm-0.in‚ 710.40625 disk.sda2.in‚0 disk.sda.in‚0 proc.cpu.threads‚17 disk.sda3.in‚0$ disk./.usageGB‚1097.590663909912 disk.nvme0c0n1.in‚ 710.40625 network.sent‚ 1.1064557e+07 disk.dm-1.out‚ 153.32421875( disk.nvme0c0n1.out‚58356.56005859375 disk.sda2.out‚0 disk.drbd1777.in‚0' proc.memory.availableMB‚ 374.8515625 disk.sda1.in‚0 disk.nvme0n1.in‚ 710.40625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚85.3125$ memory_percent‚97.71208763122559h°âÉ2- _wandbruntime‚3681  _runtime‚3681\©>øÇÁ trainer/global_step‚4899 % _timestamp‚1.7884110711210177e+09 _runtime‚3681.533149552 _step‚400 & lr-AdamW/pg1‚7.497616659956153e-05 & lr-AdamW/pg2‚7.497616659956153e-05YK„FÌ‘Æ & lr-AdamW/pg2‚7.497616659956153e-05 trainer/global_step‚4899 % _timestamp‚1.7884110711210177e+09 _runtime‚3681.533149552 _step‚400 & lr-AdamW/pg1‚7.497616659956153e-05—˜/2’- _wandbruntime‚3681  _runtime‚3681Ê›ª}Ú'“ÔO WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚145301664282624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚146624733920256 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27920369 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚153660059493888 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚401 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99169643 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 $ _timestamp‚1.788411071121766e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65713873 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6895168930344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚138537995985828 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚131642661934080 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4899 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64737291 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6915105052672 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6895334051748 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99149605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚138386559229952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040žùPr#”ÙO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚145301664282624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 163886894 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚146624757860352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6915105052672 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6895334051748 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99149605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚138386559229952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚145301664282624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚146624733920256 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27920369 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚153660059493888 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚401 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99169643 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 $ _timestamp‚1.788411071121766e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65713873 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6895168930344 ëú(ýmKEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚138537995985828 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚131642661934080 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4899 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64737291 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65726655 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6916667204096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚138413846597632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7036725877248 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6916498190848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27925774 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6896723216500 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚138413846597632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7036515305984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99169392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99169392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6916498190848 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚153690396685824 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚131668588077056 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚138413828594688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163939406 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99189456 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65727179 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6916667204096 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64749950 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99189456 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27926296 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _step‚402 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚153690162174464Q£áé&—ãM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163919340 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64749948 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚146653646868480 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚138565328332916 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚138413846597632 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7036725877248 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6916498190848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27925774 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6896723216500 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚138413846597632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7036515305984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99169392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99169392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6916498190848 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚153690396685824 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚131668588077056 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚138413828594688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163939406 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99189456 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65727179 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6916667204096 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64749950 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99189456 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27926296 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _step‚402 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚153690162174464 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚145330513801728 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚145330513801728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37800883 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65726655 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6916667204096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚146734803842048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 _step‚405 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚132939804011520 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28193994 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6984945375232 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚139750027466752 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚139750027466752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚559 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚155176514046464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚146734990844928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65370239 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚139904444760272 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100162345 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7105896093184 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139904626921036 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 `LDeviceStaœ•BŽôtsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚139750045469696 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66356909 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚561 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65370241 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100142307 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28194553 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4949 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66356348 _runtime‚ 3716.92615761 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚146734990844928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65370241 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚148070405271552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚146734803842048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 _step‚405 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚132939804011520 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28193994 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6984945375232 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚139750027466752 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚139750027466752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚559 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚155176514046464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚146734990844928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65370239 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚139904444760272 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100162345 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7105896093184 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139904626921036 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚139750045469696 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66356909 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚561 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65370241 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100142307 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28194553 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4949 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66356348 _runtime‚ 3716.92615761 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚146734990844928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65370241 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚148070405271552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚132965764233216 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165565058 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65382900 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _runtime‚3717.590332441 _step‚406 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6986169513472 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165565058 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38169731 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65382898 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66369085 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚146763466344960 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65382898 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100182158 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚148099318219776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚148099294279680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139931777107152 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚139777296831488 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7107527076864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6966029913296 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100182158 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165544992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7107318602752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚139777296831488 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65382900 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66369620 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚146763653361152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚146763653361152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38169733 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28199887 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚155206845296640 % _timestamp‚1.7884111071774132e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0ô]\Îë§èM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38169731 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65382898 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66369085 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚146763466344960 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65382898 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100182158 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚148099318219776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚148099294279680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139931777107152 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚139777296831488 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7107527076864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6966029913296 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100182158 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165544992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7107318602752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚139777296831488 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65382900 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66369620 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚146763653361152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚146763653361152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38169733 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28199887 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚155206845296640 % _timestamp‚1.7884111071774132e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28199354 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚155206612882432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚139931959268124 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6986338526720 F6DeviceStatsMonitor.o‡ªp– n_train_batch_end/segment.all.peak‚416 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚132965764233216 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165565058 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65382900 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _runtime‚3717.590332441 _step‚406 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6986169513472 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165565058–žÈÆ)2¨- _wandbruntime‚3717  _runtime‚3717^ ::Y©Ó 0loss/FRt3/assignment_loss‚2.4127766204916914 _step‚407 trainer/global_step‚4949 0loss/FBt4/assignment_loss‚0.5489616275037406 .loss/FRt2/detection_loss‚2.175169514982324 %loss/total_loss‚50.00677193182779 epoch‚4 0loss/FBt1/assignment_loss‚0.5875774488260503 1loss/SRqqt4/detection_loss‚1.9544269238032548 .loss/FBt1/detection_loss‚3.064087504148483 2loss/SRqqt2/assignment_loss‚1.3328685067475787 1loss/FBt3/assignment_loss‚0.32648548687656165 % _timestamp‚1.7884111071793742e+09 /loss/FRt4/assignment_loss‚4.080426705154506 1loss/SRqqt1/assignment_loss‚2.188265170902014 1loss/FBt2/assignment_loss‚0.43916125463955624 /loss/FRt1/detection_loss‚2.9571805315859176 1loss/SRqqt1/detection_loss‚2.6691195496491025 /loss/FRt2/assignment_loss‚3.154866421431826 2loss/SRqqt4/assignment_loss‚1.6254394933873533 .loss/FRt3/detection_loss‚2.177401708358008 /loss/FRt4/detection_loss‚2.3457160355015234 /loss/FBt4/detection_loss‚3.0769669890403746 _runtime‚3717.591547761 /loss/FRt1/assignment_loss‚4.440462477066937 2loss/SRqqt3/assignment_loss‚1.6831888590346684 1loss/SRqqt3/detection_loss‚1.2569838347760114 /loss/FBt2/detection_loss‚3.1432426836755543 0loss/SRqqt2/detection_loss‚1.634928267390009 /loss/FBt3/detection_loss‚0.7310683168547448ªïÔ^ªØ trainer/global_step‚4949 0loss/FBt4/assignment_loss‚0.5489616275037406 .loss/FRt2/detection_loss‚2.175169514982324 %loss/total_loss‚50.00677193182779 epoch‚4 0loss/FBt1/assignment_loss‚0.5875774488260503 1loss/SRqqt4/detection_loss‚1.9544269238032548 .loss/FBt1/detection_loss‚3.064087504148483 2loss/SRqqt2/assignment_loss‚1.3328685067475787 1loss/FBt3/assignment_loss‚0.32648548687656165 % _timestamp‚1.7884111071793742e+09 /loss/FRt4/assignment_loss‚4.080426705154506 1loss/SRqqt1/assignment_loss‚2.188265170902014 1loss/FBt2/assignment_loss‚0.43916125463955624 /loss/FRt1/detection_loss‚2.9571805315859176 1loss/SRqqt1/detection_loss‚2.6691195496491025 /loss/FRt2/assignment_loss‚3.154866421431826 2loss/SRqqt4/assignment_loss‚1.6254394933873533 .loss/FRt3/detection_loss‚2.177401708358008 /loss/FRt4/detection_loss‚2.3457160355015234 /loss/FBt4/detection_loss‚3.0769669890403746 _runtime‚3717.591547761 /loss/FRt1/assignment_loss‚4.440462477066937 2loss/SRqqt3/assignment_loss‚1.6831888590346684 1loss/SRqqt3/detection_loss‚1.2569838347760114 /loss/FBt2/detection_loss‚3.1432426836755543 0loss/SRqqt2/detection_loss‚1.634928267390009 /loss/FBt3/detection_loss‚0.7310683168547448 0loss/FRt3/assignment_loss‚2.4127766204916914 _step‚407—¡qV«‚(:… åùãÔŸœ™™ gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.111' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”fl(: åùãÔ¦•œ disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 155.03125 cpu‚25.3931243443998) disk./.usagePercent‚73.66023193094728 disk.sda1.in‚0 disk.drbd1777.in‚0% disk./.usageGB‚1097.5906867980957 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.out‚0 disk.sda.in‚0 network.recv‚ 1.411072e+06 disk.rbd2.out‚ 86.96484375( disk.nvme0c0n1.out‚58555.34228515625 disk.sda3.out‚ 155.03125 disk.drbd1777.out‚0" proc.memory.rssMB‚ 2878.3671875 disk.dm-1.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 710.40625& disk.nvme0n1.out‚58555.34228515625 disk.dm-0.in‚ 710.40625( proc.memory.availableMB‚ 370.32421875 proc.cpu.threads‚17 disk.rbd1.out‚ 2.05078125# disk.dm-0.out‚58555.34228515625 network.sent‚ 1.1199692e+07 disk.rbd3.in‚0 disk.sda.out‚ 155.03125* proc.memory.percent‚17.568159103393555 disk.rbd0.in‚0 disk.sda3.in‚0$ memory_percent‚97.73972034454346UDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101155047 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚141113513706496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7054447697920 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚148168130404352 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 167158238 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38531204 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚156692840887808 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚141113495703552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28467705 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7034112567160 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148167943401472 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66003191 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38531206 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7034277688564 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66003189 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101135009 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0äDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28467705 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7034112567160 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148167943401472 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66003191 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38531206 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7034277688564 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66003189 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101135009 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66003191 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚149516028742656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202195968 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4999 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101155047 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66999422 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚156693067023872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7176812145152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101155047 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚141113513706496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7054447697920 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚148168130404352 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 167158238 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38531204™)]'2·- _wandbruntime‚3752  _runtime‚3752$ËC ¸âM FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 _step‚410 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.smB4J³Hall_pool.current‚ 169013248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚141140765068288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28473217 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167170644 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚141140765068288 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101174860 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚148196792920576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167190710 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167190710 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚141140783071232 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67011798 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7178233311232 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚134262923350016 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148196605904384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66015850 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚134262906310656 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 167170644 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚148196792920576 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7055840836096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7035666853316 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66015848 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38538583 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149544941690880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚3752.972853786 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7055840836096 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚141298408042360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚141140783071232 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101154796 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7035501731704 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 trainer/global_step‚4999 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚156723415804928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149544965630976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0¢­sµí&¹çM E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167170644 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚141140765068288 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101174860 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚148196792920576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167190710 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167190710 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚141140783071232 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67011798 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7178233311232 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚134262923350016 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148196605904384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66015850 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚134262906310656 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 167170644 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚148196792920576 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7055840836096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7035666853316 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66015848 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38538583 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149544941690880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚3752.972853786 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7055840836096 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚141298408042360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚141140783071232 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101154796 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7035501731704 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 trainer/global_step‚4999 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚156723415804928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149544965630976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 _step‚410 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚141140765068288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28473217 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408š«©ÍY2º- _wandbruntime‚3752  _runtime‚3752N}±(W»Ñ /loss/FRt1/detection_loss‚3.2480488841732345 /loss/FRt2/detection_loss‚2.0888708966045546 _runtime‚ 3752.97409767 0loss/FRt1/assignment_loss‚5.4468957881132765 /loss/FRt4/assignment_loss‚3.521421192780785 2loss/SRqqt3/assignment_loss‚1.4861287149660132 0loss/FBt1/assignment_loss‚0.6577396951430604 /loss/FRt4/detection_loss‚2.0959025180858113 &loss/total_loss‚50.492215978048534 epoch‚4 1loss/FBt3/assignment_loss‚0.22076626344303585 2loss/SRqqt4/assignment_loss‚1.4891897711370672 /loss/FBt2/detection_loss‚3.0693297584851584 /loss/FRt2/assignment_loss‚2.911153661793676 /loss/FBt2/assignment_loss‚0.432295598089695 0loss/SRqqt4/detection_loss‚2.185879378446511 % _timestamp‚1.7884111425620399e+09 2loss/SRqqt2/assignment_loss‚1.3794749188128599 /loss/FRt3/detection_loss‚2.1008047616862235 1loss/SRqqt2/detection_loss‚1.3640971123998942 /loss/FBt3/detection_loss‚1.2109171051287082 /loss/FBt4/detection_loss‚2.9145208434625105 _step‚411 0loss/FBt4/assignment_loss‚0.4384486751904743 0loss/SRqqt1/detection_loss‚2.605086532660893 ,loss/FBt1/detection_loss‚2.8232449834997 /loss/FRt3/assignment_loss‚2.670231620047955 2loss/SRqqt1/assignment_loss‚2.5809410563537054 1loss/SRqqt3/detection_loss‚1.5508262475437307 trainer/global_step‚4999M]Z\¼Ö 0loss/FRt1/assignment_loss‚5.4468957881132765 /loss/FRt4/assignment_loss‚3.521421192780785 2loss/SRqqt3/assignment_loss‚1.4861287149660132 0loss/FBt1/assignment_loss‚0.6577396951430604 /loss/FRt4/detection_loss‚2.0959025180858113 &loss/total_loss‚50.492215978048534 epoch‚4 1loss/FBt3/assignment_loss‚0.22076626344303585 2loss/SRqqt4/assignment_loss‚1.4891897711370672 /loss/FBt2/detection_loss‚3.0693297584851584 /loss/FRt2/assignment_loss‚2.911153661793676 /loss/FBt2/assignment_loss‚0.432295598089695 0loss/SRqqt4/detection_loss‚2.185879378446511 % _timestamp‚1.7884111425620399e+09 2loss/SRqqt2/assignment_loss‚1.3794749188128599 /loss/FRt3/detection_loss‚2.1008047616862235 1loss/SRqqt2/detection_loss‚1.3640971123998942 /loss/FBt3/detection_loss‚1.2109171051287082 /loss/FBt4/detection_loss‚2.9145208434625105 _step‚411 0loss/FBt4/assignment_loss‚0.4384486751904743 0loss/SRqqt1/detection_loss‚2.605086532660893 ,loss/FBt1/detection_loss‚2.8232449834997 /loss/FRt3/assignment_loss‚2.670231620047955 2loss/SRqqt1/assignment_loss‚2.5809410563537054 1loss/SRqqt3/detection_loss‚1.5508262475437307 trainer/global_step‚4999 /loss/FRt1/detection_loss‚3.2480488841732345 /loss/FRt2/detection_loss‚2.0888708966045546 _runtime‚ 3752.97409767›ËÖøø‘½‚(:† ’úãÔ„¤™ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.273' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4546 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}hMÛ%¾‚(:š ’úãÔ¥¤¢œ disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚58754.75244140625 disk.nvme0n1.in‚ 710.40625 network.recv‚ 1.432044e+06 disk.dm-1.out‚ 156.4921875 disk.sda3.out‚ 156.4921875" memory_percent‚97.739577293396" proc.memory.rssMB‚ 2878.3671875* proc.memory.percent‚17.568159103393555 network.sent‚ 1.132234e+07 disk.sda.out‚ 156.4921875 disk.drbd1777.out‚0 disk.rbd1.out‚ 2.14599609375( proc.memory.availableMB‚ 370.34765625 disk.nvme0c0n1.in‚ 710.40625 disk.dm-1.in‚0 disk.rbd2.out‚87.20361328125 disk.sda.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 710.40625% disk./.usageGB‚1097.5906677246094( disk.nvme0c0n1.out‚58754.75244140625 disk.sda2.out‚0 disk.sda3.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0( disk./.usagePercent‚73.6602306509096 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda2.in‚0# disk.dm-0.out‚58754.75244140625 disk.sda1.in‚0 cpu‚25.389048460895808\®OÒ¿‚(:… ¡úãÔ’¹­™ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚91.613' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.3226 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N5+À‚(:  ¡úãÔÇ€œ disk.nvme0c0n1.in‚ 710.40625( disk.nvme0c0n1.out‚58755.26611328125 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.14599609375# disk.dm-0.out‚58755.26611328125" proc.memory.rssMB‚ 2878.3671875 disk.sda.in‚0 disk.sda.out‚ 157.12890625 disk.sda1.out‚0 disk.drbd1777.out‚0' proc.memory.availableMB‚ 246.3046875 network.sent‚ 1.1437823e+07 disk.dm-1.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 157.12890625 disk.rbd0.out‚0 cpu‚25.387715456803967 disk.rbd0.in‚0 disk.dm-0.in‚ 710.40625$ memory_percent‚98.49667549133301 disk.rbd3.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.rbd2.out‚87.20361328125 disk.sda3.in‚0* proc.memory.percent‚17.568159103393555 disk.nvme0n1.in‚ 710.40625& disk.nvme0n1.out‚58755.26611328125 network.recv‚ 1.452557e+06) disk./.usagePercent‚73.66019993000549% disk./.usageGB‚1097.5902099609375 disk.sda2.out‚0 disk.sda3.out‚ 157.12890625 disk.drbd1777.in‚0 disk.rbd2.in‚0¾~ñ‰2Á- _wandbruntime‚3787  _runtime‚3787÷K’÷Å¿ trainer/global_step‚5049 % _timestamp‚1.7884111771834779e+09 _runtime‚3787.595613955 _step‚412 % lr-AdamW/pg1‚7.30395328566305e-05 % lr-AdamW/pg2‚7.30395328566305e-05ã_=ÊÃÄ trainer/global_step‚5049 % _timestamp‚1.7884111771834779e+09 _runtime‚3787.595613955 _step‚412 % lr-AdamW/pg1‚7.30395328566305e-05 % lr-AdamW/pg2‚7.30395328566305e-05œ"çZV2Ä- _wandbruntime‚3787  _runtime‚3787L°S= "ÅÙO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚149601082960896 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚150961676153856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 % _timestamp‚1.7884111771841874e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚142476963940352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7124288020480 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚158209589970432 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚150961700093952 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚142637706630688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7124119020544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚142476981943296 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚149601269963776 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102147749 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102147749 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7124119020544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚510 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚508 _runtime‚3787.596587099 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66636139 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102127711 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28740092 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 trainer/global_step‚5049 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚142476963940352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7247889876480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚135534122245120 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚142637888791452 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66636141 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66636139 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67640148 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚149601082960896 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚149601269963776 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168783890 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7247677194752 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚135534139284480 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158209353348608 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38900054 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66636141 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _step‚413 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7103749506972 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38900056 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102127711 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0§Wã_ä'ÆÞO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66636141 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _step‚413 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7103749506972 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38900056 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102127711 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚149601082960896 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚150961676153856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 % _timestamp‚1.7884111771841874e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚142476963940352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7124288020480 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚158209589970432 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚150961700093952 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚142637706630688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7124119020544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚142476981943296 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚149601269963776 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102147749 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102147749 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7124119020544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚510 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚508 _runtime‚3787.596587099 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66636139 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102127711 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28740092 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 trainer/global_step‚5049 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚142476963940352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7247889876480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚135534122245120 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚142637888791452 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66636141 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66636139 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67640148 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚149601082960896 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚149601269963776 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168783890 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7247677194752 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚135534139284480 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158209353348608 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38900054 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160æýãð2Ç- _wandbruntime‚3788  _runtime‚3788ì …cé&ÈãM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223154176 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚135560065427456 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142504251308032 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚150990589102080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7105138671724 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102147498 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚142665038977568 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _runtime‚3788.250709526 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 trainer/global_step‚5049 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168796296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38907431 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 247094272 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚158239673124864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38907433 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168816362 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚135560082466816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚149629932480000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28744845 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 % _timestamp‚1.7884111778378842e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66648800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66648800 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66648798 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7249307176960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _step‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7125681171968 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102167562 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚149629745463808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102147498 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚142504251308032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7125512158720 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102167562 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 168796296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 168816362 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7125681171968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28745379 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚158239920219136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7104973550112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67652276 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612HbXî&ÉèM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚142504233305088 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚149629932480000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67652276 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223154176 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚135560065427456 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142504251308032 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚150990589102080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7105138671724 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102147498 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚142665038977568 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _runtime‚3788.250709526 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 trainer/global_step‚5049 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168796296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38907431 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 247094272 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚158239673124864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38907433 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168816362 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚135560082466816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚149629932480000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28744845 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 % _timestamp‚1.7884111778378842e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66648800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66648800 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66648798 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7249307176960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _step‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7125681171968 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102167562 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚149629745463808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102147498 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚142504251308032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7125512158720 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102167562 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 168796296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 168816362 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7125681171968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28745379 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚158239920219136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7104973550112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68282973 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7193790343168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7193790343168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67269089 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚159725850062336 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚136831298401280 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚3822.520407617 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170389502 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚151034409523200 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67269091 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚151034222520320 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7318526497280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67269089 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚151034409523200 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚563 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚159726084587008 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚144004519726660 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 trainer/global_step‚5099 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170409542 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103140451 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7173056203976 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103120413 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7193959343104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7193959343104 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚152407347505152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚136831281361920 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚143840432177152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚144004337565896 $ _timestamp‚1.788411212108098e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68282410 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67269091 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚143840450180096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29014067 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚152407323565056 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7318737081856 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚143840432177152 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7173221325380 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199B|œã'ÖÝO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚136831281361920 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚143840432177152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚144004337565896 $ _timestamp‚1.788411212108098e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68282410 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67269091 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚143840450180096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29014067 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚152407323565056 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7318737081856 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚143840432177152 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7173221325380 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68282973 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7193790343168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7193790343168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67269089 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚159725850062336 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚136831298401280 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚3822.520407617 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170389502 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚151034409523200 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67269091 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚151034222520320 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7318526497280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67269089 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚151034409523200 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚563 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚159726084587008 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚144004519726660 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 trainer/global_step‚5099 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170409542 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103140451 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7173056203976 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103120413 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7193959343104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7193959343104 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚152407347505152¡—Üã!2×- _wandbruntime‚3823  _runtime‚3823ÛeÙ´„ØâM PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7195183481344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29019459 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 trainer/global_step‚5099 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚143867701541888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39276281 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7195352494592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170421948 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103160264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29018924 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7320155398144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67281750 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚144031669912776 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67281748 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚143867719544832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚ 3823.18691357 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68295205 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7319942729728 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7174445368520 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7174610490132 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _step‚418 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68295742 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚159756179243008 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚136857224544256 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 170421948 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7195352494592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚151063072039424 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159756415851520 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚136857241583616 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚152436260453376 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170442014 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67281748 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39276283 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170442014 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67281748 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39276283 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7195183481344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29019459 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 trainer/global_step‚5099 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚143867701541888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39276281 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7195352494592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170421948 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103160264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29018924 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7320155398144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67281750 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚144031669912776 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67281748 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚143867719544832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚ 3823.18691357 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68295205 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7319942729728 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7174445368520 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7174610490132 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _step‚418 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68295742 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚159756179243008 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚136857224544256 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 170421948 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7195352494592 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚151063072039424 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159756415851520 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚136857241583616 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160¢(¯_2Ú- _wandbruntime‚3823  _runtime‚3823K¢ÝUQÛË /loss/FRt1/assignment_loss‚5.102807111740113 0loss/FRt3/assignment_loss‚1.9329485906494988 1loss/FBt2/assignment_loss‚0.24304124154150486 .loss/FRt1/detection_loss‚3.151518840789795 -loss/FRt2/detection_loss‚2.11901476373896 _step‚419 0loss/FBt1/assignment_loss‚0.6196129699790618 epoch‚4 0loss/FRt4/assignment_loss‚3.6335230119850324 -loss/FBt1/detection_loss‚3.10023832321167 % _timestamp‚1.7884112127760153e+09 0loss/FRt2/assignment_loss‚2.6023261696100235 0loss/SRqqt1/detection_loss‚2.734260303358878 .loss/FBt4/assignment_loss‚0.48528076697169 /loss/FRt3/detection_loss‚2.0661588130725757 1loss/SRqqt2/detection_loss‚1.4600854562474537 /loss/FBt3/detection_loss‚0.7535274373262352 %loss/total_loss‚48.83909844444956 _runtime‚3823.188069499 1loss/SRqqt1/assignment_loss‚1.987921005295169 /loss/FBt2/detection_loss‚3.5662054121494293 1loss/SRqqt4/assignment_loss‚1.597883868102844 1loss/SRqqt4/detection_loss‚2.0243958891966405 2loss/SRqqt2/assignment_loss‚1.7473934493571908 1loss/SRqqt3/assignment_loss‚1.661656164060362 1loss/FBt3/assignment_loss‚0.26720975136730496 .loss/FRt4/detection_loss‚2.142002866967865 0loss/SRqqt3/detection_loss‚1.474204460494831 -loss/FBt4/detection_loss‚2.36588177723544 trainer/global_step‚5099÷GëVÜÐ 0loss/FRt2/assignment_loss‚2.6023261696100235 0loss/SRqqt1/detection_loss‚2.734260303358878 .loss/FBt4/assignment_loss‚0.48528076697169 /loss/FRt3/detection_loss‚2.0661588130725757 1loss/SRqqt2/detection_loss‚1.4600854562474537 /loss/FBt3/detection_loss‚0.7535274373262352 %loss/total_loss‚48.83909844444956 _runtime‚3823.188069499 1loss/SRqqt1/assignment_loss‚1.987921005295169 /loss/FBt2/detection_loss‚3.5662054121494293 1loss/SRqqt4/assignment_loss‚1.597883868102844 1loss/SRqqt4/detection_loss‚2.0243958891966405 2loss/SRqqt2/assignment_loss‚1.7473934493571908 1loss/SRqqt3/assignment_loss‚1.661656164060362 1loss/FBt3/assignment_loss‚0.26720975136730496 .loss/FRt4/detection_loss‚2.142002866967865 0loss/SRqqt3/detection_loss‚1.474204460494831 -loss/FBt4/detection_loss‚2.36588177723544 trainer/global_step‚5099 /loss/FRt1/assignment_loss‚5.102807111740113 0loss/FRt3/assignment_loss‚1.9329485906494988 1loss/FBt2/assignment_loss‚0.24304124154150486 .loss/FRt1/detection_loss‚3.151518840789795 -loss/FRt2/detection_loss‚2.11901476373896 _step‚419 0loss/FBt1/assignment_loss‚0.6196129699790618 epoch‚4 0loss/FRt4/assignment_loss‚3.6335230119850324 -loss/FBt1/detection_loss‚3.10023832321167 % _timestamp‚1.7884112127760153e+09£ùO(Š‘Ý‚(:† ÎúãÔÞ¦©™ gpu.0.gpu‚20.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6726 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚ª)Þ‚(:ž ÎúãÔ©©Ù› disk.nvme0n1.in‚ 710.40625 disk.dm-0.in‚ 710.40625 disk.rbd3.out‚0 disk.drbd1777.out‚0& disk.nvme0n1.out‚58954.44873046875 disk.sda3.in‚0 disk.sda3.out‚ 158.74609375 disk.sda1.out‚0 disk.rbd1.in‚0' proc.memory.availableMB‚ 246.5546875" proc.memory.rssMB‚ 2878.3671875 disk.rbd2.out‚88.93408203125( disk.nvme0c0n1.out‚58954.44873046875$ memory_percent‚98.49514961242676% disk./.usageGB‚1097.5554466247559 disk.dm-1.out‚ 158.74609375 disk.rbd2.in‚0 disk.sda.out‚ 158.74609375 disk.drbd1777.in‚0 network.sent‚ 1.1557344e+07 network.recv‚ 1.467091e+06 disk.rbd1.out‚ 2.208984375) disk./.usagePercent‚73.65786693334596# disk.dm-0.out‚58954.44873046875 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda1.in‚0* proc.memory.percent‚17.568159103393555 disk.sda2.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 cpu‚25.383771914378535 proc.cpu.threads‚17ä$ü‘ß‚(:† ÝúãÔ‡¡­™ gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.011' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0022 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄÙ_ñ à‚(:• ÝúãԧЋœ disk.rbd3.out‚0 network.sent‚ 1.1665762e+07 disk.dm-0.in‚ 710.40625 disk.rbd2.in‚0 disk.rbd2.out‚88.93408203125* proc.memory.percent‚17.568159103393555 proc.cpu.threads‚17& disk.nvme0n1.out‚59053.80126953125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚59053.80126953125 disk.sda3.in‚0( proc.memory.availableMB‚ 246.01953125 disk.dm-1.in‚0 network.recv‚ 1.473257e+06% disk./.usageGB‚1097.5554847717285 disk.nvme0c0n1.in‚ 710.40625 disk.drbd1777.out‚0 disk.dm-1.out‚ 159.21875 disk.sda.out‚ 159.21875 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.sda3.out‚ 159.21875 disk.rbd0.out‚0( disk./.usagePercent‚73.6578694934213 disk.rbd1.in‚0 disk.rbd1.out‚ 2.208984375# disk.dm-0.out‚59053.80126953125 disk.rbd3.in‚0 disk.nvme0n1.in‚ 710.40625" proc.memory.rssMB‚ 2878.3671875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.in‚0$ memory_percent‚98.49841594696045 cpu‚25.382471035222384‚šRÓ‘á‚(:† ìúãÔñç®™ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.398' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0us8'â‚(:’ ìúãÔ€´ôœ disk.sda3.out‚ 159.87109375 disk.rbd1.out‚ 2.208984375 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.sda.out‚ 159.87109375 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 710.40625# disk.nvme0n1.out‚59153.08203125* proc.memory.percent‚17.568159103393555) disk./.usagePercent‚73.65787947771514$ memory_percent‚98.48613739013672 cpu‚25.381174577703955 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda.in‚0 disk.dm-0.out‚59153.08203125 disk.dm-1.out‚ 159.87109375 disk.rbd3.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 2878.3671875 proc.cpu.threads‚17 disk.rbd2.in‚0% disk.nvme0c0n1.out‚59153.08203125 disk.sda3.in‚0% proc.memory.availableMB‚ 248.03125 network.sent‚ 1.1671084e+07 network.recv‚ 1.477128e+06% disk./.usageGB‚1097.5556335449219 disk.rbd2.out‚ 89.0986328125 disk.dm-0.in‚ 710.406257‡l2ã- _wandbruntime‚3857  _runtime‚3857ê«ÙÃä½ # _timestamp‚1.78841124753517e+09 _runtime‚ 3857.9473549 _step‚420 & lr-AdamW/pg1‚7.172475152669609e-05 & lr-AdamW/pg2‚7.172475152669609e-05 trainer/global_step‚5149?ý”¼Èå trainer/global_step‚5149 # _timestamp‚1.78841124753517e+09 _runtime‚ 3857.9473549 _step‚420 & lr-AdamW/pg1‚7.172475152669609e-05 & lr-AdamW/pg2‚7.172475152669609e-05¤®,³2æ- _wandbruntime‚3857  _runtime‚3857ÓJ­ß'çÙO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 172015154 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚152467549082624 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7263461665792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39637754 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152467362079744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145203918416896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39637756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29285177 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚145371150661868 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7263630665728 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7263461665792 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104113115 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 trainer/global_step‚5149 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚145203900413952 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67902039 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7242693143788 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _step‚421 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67902041 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 _runtime‚3857.948352209 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161242376850432 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚145370968501104 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚138128440478720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104133153 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161242615569408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7242528022384 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104133153 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104113115 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944o¢CxèÞO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39637756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29285177 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚145371150661868 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7263630665728 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7263461665792 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104113115 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 trainer/global_step‚5149 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚145203900413952 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67902039 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7242693143788 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _step‚421 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67902041 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 _runtime‚3857.948352209 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161242376850432 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚145370968501104 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚138128440478720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104133153 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161242615569408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7242528022384 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104133153 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104113115 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 172015154 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚152467549082624 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7263461665792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39637754 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152467362079744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145203918416896¥Èoä2é- _wandbruntime‚3858  _runtime‚3858 º&~é&êãM EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7264854803968 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68935830 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 % _timestamp‚1.7884112481826906e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚145231169778688 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚152496211598848 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104132902 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244997120 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7391037606400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚138154400700416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39645131 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚145231187781632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29290699 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚153881907864576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172067666 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67914700 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104152966 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161272700473856 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 172047600 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104152966 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0§Ìý(î&ëèM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 221057024 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7264854803968 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68935830 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 % _timestamp‚1.7884112481826906e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚145231169778688 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚152496211598848 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104132902 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244997120 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7391037606400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚138154400700416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39645131 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚145231187781632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29290699 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚153881907864576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172067666 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67914700 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104152966 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161272700473856 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 172047600 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104152966 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 % _timestamp‚1.7884112829484575e+09 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚146737599436312 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68534989 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚153900501639168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚139425616634880 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68534991 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚146567368650752 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68534989 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7333301988352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚162759193494528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7312164962196 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚153900501639168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚146737781597076 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105125855 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7333301988352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚155298642327552 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69564591 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚3893.360742037 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚153900688642048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40006604 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚139425599595520 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚153900688642048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7333132988416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29558539 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173640806 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105105817 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚146567368650752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0{”±œä'øÞO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105105817 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚146567386653696 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40006606 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173660846 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7311999840792 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40006604 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚139425599595520 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚153900688642048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7333132988416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29558539 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173640806 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105105817 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚146567368650752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 % _timestamp‚1.7884112829484575e+09 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚146737599436312 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68534989 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚153900501639168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚139425616634880 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68534991 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚146567368650752 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68534989 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7333301988352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚162759193494528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7312164962196 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚153900501639168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚146737781597076 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105125855 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7333301988352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚155298642327552 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69564591 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚3893.360742037 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚153900688642048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚139451559817216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚153929164142080 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105145668 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7313389005336 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚153929351158272 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚139451542777856 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚146765113944164 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173693318 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69577350 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68547648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚146594656018432 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚155327531335680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105125604 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚146594638015488 _step‚426 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68547648 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 % _timestamp‚1.7884112835920546e+09 _runtime‚3894.004760562 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105125604 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚155327555275776 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7334695139840 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68547650 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69577877­‹oÂûèM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 trainer/global_step‚5199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_bÓͤ,%atch_end/inactive_split.large_pool.freed‚40013981 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29563369 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7334695139840 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚146594638015488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7313554126948 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚162789280089600 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105125604 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚146594638015488 _step‚426 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68547648 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 % _timestamp‚1.7884112835920546e+09 _runtime‚3894.004760562 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105125604 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚155327555275776 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7334695139840 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68547650 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69577877 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173693318 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚153929351158272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29563894 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚139451559817216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚153929164142080 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105145668 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7313389005336 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚153929351158272 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚139451542777856 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚146765113944164 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173693318 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69577350 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68547648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚146594656018432 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚155327531335680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 Fm<¥Ì…Æ & lr-AdamW/pg1‚7.039229278633612e-05 & lr-AdamW/pg2‚7.039229278633612e-05 trainer/global_step‚5249 % _timestamp‚1.7884113178751419e+09 _runtime‚3928.287345141 _step‚428¬|þÉ 2†- _wandbruntime‚3928  _runtime‚3928@£‚×ß'‡ÙO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29831650 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69167941 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226136064 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70206582 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70207106 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚147930854890496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7402804311040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7381636780604 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚147930836887552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202195968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69167939 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚155333828201472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚155333641198592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884113178759289e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106098519 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7402973310976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40375454 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7531437688320 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7381471659200 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚164275501291008 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69167939 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 _runtime‚3928.288258362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175286498 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106098519 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175266458 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚148104230371520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40375456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164275727427072 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚524 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚156744265798656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚140722758712320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚156744289738752 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7402804311040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70207106 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚147930854890496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7402804311040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7381636780604 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚147930836887552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202195968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69167939 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚155333828201472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ?9DeviceStatsMonitor.on_trainE\~F£ _batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚155333641198592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884113178759289e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106098519 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7402973310976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40375454 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7531437688320 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7381471659200 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚164275501291008 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69167939 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 _runtime‚3928.288258362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175286498 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106098519 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175266458 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚148104230371520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40375456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164275727427072 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚524 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚156744265798656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚140722758712320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚156744289738752 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969­™RD·2‰- _wandbruntime‚3928  _runtime‚3928oµ é&ŠãM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70219618 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _runtime‚3928.941396386 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175318970 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7382860823744 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚550 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7532861689856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚140748701894656 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69180598 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175318970 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106118306 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7383025945356 % _timestamp‚1.7884113185287907e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚147958106252288 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69180600 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚156773202686976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29836787 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69180600 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106138370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29837335 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚155362303701504 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175298904 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚5249 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7404197449216 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚156773178746880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 _step‚430 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106118306 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70220168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚548 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚148131562718400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160y¡-úî&‹èM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29837335 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚155362303701504 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175298904 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚5249 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7404197449216 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚156773178746880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 _step‚430 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106118306 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70220168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚548 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚148131562718400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70219618 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _runtime‚3928.941396386 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175318970 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7382860823744 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚550 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7532861689856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚140748701894656 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69180598 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175318970 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106118306 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7383025945356 % _timestamp‚1.7884113185287907e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚147958106252288 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69180600 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚156773202686976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29836787 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69180600 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106138370®Ëïh2Œ- _wandbruntime‚3928  _runtime‚3928Ý&ù[Õ 0loss/FRt4/assignment_loss‚3.6270967260846554 2loss/SRqqt2/assignment_loss‚1.3002511679797488 .loss/FRt4/detection_loss‚2.149613362438274 /loss/FRt1/assignment_loss‚4.046207631787946 2loss/SRqqt3/assignment_loss‚1.3205824112709688 /loss/FRt3/detection_loss‚2.2818045428333185 /loss/FBt3/detection_loss‚0.8883129781549997 %loss/total_loss‚46.65352275811274 1loss/SRqqt4/assignment_loss‚1.477159154035297 /loss/FBt1/detection_loss‚2.7220809001189012 % _timestamp‚1.7884113185309384e+09 _step‚431 0loss/FBt1/assignment_loss‚0.7380591692143263 1loss/FBt4/assignment_loss‚0.44672884670983065 1loss/SRqqt2/detection_loss‚1.7551581186845022 0loss/FBt2/assignment_loss‚0.3693511851131916 .loss/FRt2/detection_loss‚2.138053603172302 1loss/SRqqt4/detection_loss‚1.8640154968289768 /loss/FBt2/detection_loss‚2.4604005932807924 /loss/FBt4/detection_loss‚2.1511888754995248 epoch‚4 _runtime‚3928.943004206 0loss/FRt3/assignment_loss‚2.3895548383394876 /loss/FRt1/detection_loss‚2.8978625305237307 2loss/SRqqt1/assignment_loss‚1.9985613177220027 0loss/FBt3/assignment_loss‚0.2658727870898083 0loss/SRqqt1/detection_loss‚2.568123439947764 1loss/SRqqt3/detection_loss‚1.4651330739867932 trainer/global_step‚5249 0loss/FRt2/assignment_loss‚3.3323500072956085à½!`ŽÚ 1loss/SRqqt4/assignment_loss‚1.477159154035297 /loss/FBt1/detection_loss‚2.7220809001189012 % _timestamp‚1.7884113185309384e+09 _step‚431 0loss/FBt1/assignment_loss‚0.7380591692143263 1loss/FBt4/assignment_loss‚0.44672884670983065 1loss/SRqqt2/detection_loss‚1.7551581186845022 0loss/FBt2/assignment_loss‚0.3693511851131916 .loss/FRt2/detection_loss‚2.138053603172302 1loss/SRqqt4/detection_loss‚1.8640154968289768 /loss/FBt2/detection_loss‚2.4604005932807924 /loss/FBt4/detection_loss‚2.1511888754995248 epoch‚4 _runtime‚3928.943004206 0loss/FRt3/assignment_loss‚2.3895548383394876 /loss/FRt1/detection_loss‚2.8978625305237307 2loss/SRqqt1/assignment_loss‚1.9985613177220027 0loss/FBt3/assignment_loss‚0.2658727870898083 0loss/SRqqt1/detection_loss‚2.568123439947764 1loss/SRqqt3/detection_loss‚1.4651330739867932 trainer/global_step‚5249 0loss/FRt2/assignment_loss‚3.3323500072956085 0loss/FRt4/assignment_loss‚3.6270967260846554 2loss/SRqqt2/assignment_loss‚1.3002511679797488 .loss/FRt4/detection_loss‚2.149613362438274 /loss/FRt1/assignment_loss‚4.046207631787946 2loss/SRqqt3/assignment_loss‚1.3205824112709688 /loss/FRt3/detection_loss‚2.2818045428333185 /loss/FBt3/detection_loss‚0.8883129781549997 %loss/total_loss‚46.65352275811274¯±Øš‘‚(:† ·ûãÔ­à±™ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚108.734' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7468 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oóLK ‚(:• ·ûãÔ–Íž) disk./.usagePercent‚73.58664154119975 disk.nvme0c0n1.in‚ 710.40625& disk.nvme0c0n1.out‚59520.888671875 disk.rbd1.out‚ 2.208984375 disk.dm-0.in‚ 710.40625% disk./.usageGB‚1096.4941368103027 disk.rbd2.out‚90.85302734375 disk.sda3.in‚0 disk.sda.out‚ 162.5546875 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.sda.in‚0! disk.dm-0.out‚59520.888671875 disk.dm-1.out‚ 162.5546875# proc.memory.rssMB‚ 2878.37109375* proc.memory.percent‚17.568182945251465 cpu‚25.37495261586482$ disk.nvme0n1.out‚59520.888671875' proc.memory.availableMB‚ 247.6328125 network.recv‚ 1.499925e+06 disk.rbd3.out‚0 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 710.40625 disk.rbd0.out‚0 disk.sda2.in‚0 network.sent‚ 1.1903626e+07 disk.rbd1.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 162.5546875$ memory_percent‚98.48856925964355Rý?Öœ‘‚(:‘ ÆûãÔÃÜ¢™ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.395' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.078999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0výN¨$’‚(:™ ÆûãÔËïÚ disk.rbd1.in‚0 disk.dm-0.in‚ 710.40625 disk.rbd3.out‚0 disk.sda3.out‚ 163.01953125 disk.sda.in‚0# memory_percent‚98.4886884689331% disk./.usageGB‚1096.4941368103027 disk.sda2.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 247.61328125 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 710.40625 cpu‚25.373718402240566 proc.cpu.threads‚17) disk./.usagePercent‚73.58664154119975$ disk.nvme0n1.out‚59521.220703125& disk.nvme0c0n1.out‚59521.220703125 disk.sda1.in‚0* proc.memory.percent‚17.568182945251465 disk.drbd1777.out‚0! disk.dm-0.out‚59521.220703125 disk.dm-1.in‚0 network.sent‚ 1.2012204e+07 network.recv‚ 1.506554e+06 disk.rbd2.out‚90.99365234375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd1.out‚ 2.208984375 disk.dm-1.out‚ 163.01953125 disk.nvme0c0n1.in‚ 710.40625 disk.sda.out‚ 163.01953125# proc.memory.rssMB‚ 2878.37109375Skõx“‚(:„ ÕûãÔÏÞÅ™ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚93.295' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.659 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07P/*”‚(:Ÿ ÕûãÔ’¡ã›# proc.memory.rssMB‚ 2878.37109375& disk.nvme0n1.out‚59521.89794921875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚59521.89794921875 disk.rbd1.in‚0 disk.rbd2.out‚90.99365234375 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 1.510861e+06) disk./.usagePercent‚73.58896045744491 disk.dm-1.out‚ 163.60546875 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda.out‚ 163.60546875* proc.memory.percent‚17.568182945251465 network.sent‚ 1.201827e+07 disk.dm-0.in‚ 710.40625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 710.40625 cpu‚25.372525346834408 proc.cpu.threads‚17 disk.dm-1.in‚0$ memory_percent‚98.48873615264893 disk.rbd1.out‚ 2.208984375 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 163.60546875 disk.drbd1777.in‚0% disk./.usageGB‚1096.5286903381348 disk.nvme0n1.in‚ 710.40625 disk.sda2.in‚0 disk.drbd1777.out‚0( proc.memory.availableMB‚ 247.60546875# disk.dm-0.out‚59521.89794921875 disk.sda2.out‚0 disk.sda3.in‚0:ÖÚ 2•- _wandbruntime‚3963  _runtime‚3963ÒòZMÅ–¿ % lr-AdamW/pg1‚6.97197809218158e-05 % lr-AdamW/pg2‚6.97197809218158e-05 trainer/global_step‚5299 % _timestamp‚1.7884113527537675e+09 _runtime‚3963.165912616 _step‚432ãT£ìÊ—Ä % lr-AdamW/pg1‚6.97197809218158e-05 % lr-AdamW/pg2‚6.97197809218158e-05 trainer/global_step‚5299 % _timestamp‚1.7884113527537675e+09 _runtime‚3963.165912616 _step‚432°¸"–s2˜- _wandbruntime‚3963  _runtime‚3963e%ÏýÌ™ÙO WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40744306 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚149294323127296 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚149470861306728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚524 D:DeviceStatsMonitor.on_traa$2#"in_batch_start/active.all.current‚20040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156766780758016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107111259 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156766780758016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69800889 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚149294305124352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40744304 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚149471043467492 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107091221 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7472644633600 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚158189937149952 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107111259 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚165791997240832 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107091221 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚149294323127296 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69800891 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7451108599012 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 _runtime‚3963.166858899 _step‚433 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7472644633600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 176912150 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 % _timestamp‚1.7884113527545054e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚156766967760896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30106624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚156766967760896 trainer/global_step‚5299 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7602084030976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7602292518400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7602084030976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7602292518400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156766780758016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107111259 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156766780758016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69800889 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚149294305124352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40744304 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚149471043467492 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107091221 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7472644633600 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚158189937149952 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107111259 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚165791997240832 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107091221±ÌU92›- _wandbruntime‚3963  _runtime‚3963›×é&œãM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7473868771840 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107111008 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚149498375814580 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚149321574489088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7474037785088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69813548 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70863092 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107131072 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7473868771840 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69813550 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚158218850098176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7603714698240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884113533965042e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 176944622 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107111008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _step‚434 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69813550 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚142045861011456 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚158218826158080 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚142045878050816 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚156795443260928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30111940 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30111411 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚149321592492032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40751681 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 176924556 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7452497763764 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5299 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚165822564796416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚156795630277120 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156795630277120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40751683 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176944622 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107131072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7452332642152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69813548 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚149498193653608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚149321574489088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70863623 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26àfËèM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 176924556 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7452497763764 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5299 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚165822564796416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚156795630277120 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156795630277120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40751683 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176944622 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107131072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7452332642152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69813548 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚149498193653608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚149321574489088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70863623 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7473868771840 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107111008 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚149498375814580 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚149321574489088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_poÒ¹‡ #ol.allocated‚ 7474037785088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69813548 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70863092 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107131072 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7473868771840 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69813550 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚158218850098176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7603714698240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884113533965042e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 176944622 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107111008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _step‚434 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69813550 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚142045861011456 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚158218826158080 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚142045878050816 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚156795443260928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30111940 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30111411 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚149321592492032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40751681 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0²P̤æ2ž- _wandbruntime‚3963  _runtime‚3963}¾¦†XŸÒ epoch‚4 1loss/SRqqt1/detection_loss‚3.3717472553253174 &loss/total_loss‚51.389153098243554 /loss/FRt1/assignment_loss‚4.853919833047049 1loss/SRqqt1/assignment_loss‚2.116286825607805 1loss/FBt2/assignment_loss‚0.07001329440390691 /loss/FRt2/detection_loss‚1.9602521261343588 0loss/SRqqt2/detection_loss‚1.331009030705545 0loss/SRqqt4/detection_loss‚2.001470133345178 /loss/FRt3/detection_loss‚2.3729644564813688 /loss/FRt4/detection_loss‚2.3435679735506283 /loss/FBt3/detection_loss‚0.7654981502400604 /loss/FBt4/detection_loss‚2.9851982465812137 _step‚435 /loss/FRt2/assignment_loss‚2.403946219957792 2loss/SRqqt3/assignment_loss‚1.5399245809142788 0loss/FBt1/assignment_loss‚0.3476721798079476 0loss/FRt3/assignment_loss‚1.9113912770622654 2loss/SRqqt2/assignment_loss‚1.3638948401572502 .loss/FRt1/detection_loss‚2.856559269768851 0loss/SRqqt3/detection_loss‚1.829889979834358 .loss/FBt1/detection_loss‚2.682295306523641 % _timestamp‚1.7884113533981476e+09 2loss/SRqqt4/assignment_loss‚1.9089701422985563 /loss/FBt4/assignment_loss‚0.609496132788081 0loss/FRt4/assignment_loss‚3.6077249768902275 0loss/FBt3/assignment_loss‚0.2317148770183722 /loss/FBt2/detection_loss‚5.9237459897994995 trainer/global_step‚5299 _runtime‚3963.810491385“¤²] × 1loss/SRqqt1/detection_loss‚3.3717472553253174 &loss/total_loss‚51.389153098243554 /loss/FRt1/assignment_loss‚4.853919833047049 1loss/SRqqt1/assignment_loss‚2.116286825607805 1loss/FBt2/assignment_loss‚0.07001329440390691 /loss/FRt2/detection_loss‚1.9602521261343588 0loss/SRqqt2/detection_loss‚1.331009030705545 0loss/SRqqt4/detection_loss‚2.001470133345178 /loss/FRt3/detection_loss‚2.3729644564813688 /loss/FRt4/detection_loss‚2.3435679735506283 /loss/FBt3/detection_loss‚0.7654981502400604 /loss/FBt4/detection_loss‚2.9851982465812137 _step‚435 /loss/FRt2/assignment_loss‚2.403946219957792 2loss/SRqqt3/assignment_loss‚1.5399245809142788 0loss/FBt1/assignment_loss‚0.3476721798079476 0loss/FRt3/assignment_loss‚1.9113912770622654 2loss/SRqqt2/assignment_loss‚1.3638948401572502 .loss/FRt1/detection_loss‚2.856559269768851 0loss/SRqqt3/detection_loss‚1.829889979834358 .loss/FBt1/detection_loss‚2.682295306523641 % _timestamp‚1.7884113533981476e+09 2loss/SRqqt4/assignment_loss‚1.9089701422985563 /loss/FBt4/assignment_loss‚0.609496132788081 0loss/FRt4/assignment_loss‚3.6077249768902275 0loss/FBt3/assignment_loss‚0.2317148770183722 /loss/FBt2/detection_loss‚5.9237459897994995 trainer/global_step‚5299 _runtime‚3963.810491385 epoch‚4³]sd‘¡‚(:† äûãÔ𘿙 gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.896' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨Š`C#¢‚(:˜ äûãÔï謜 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.dm-0.in‚ 710.40625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda1.in‚0 cpu‚25.37130820275134 disk.rbd3.out‚0( disk.nvme0c0n1.out‚59557.41748046875 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 164.203125 disk.sda.in‚0 disk.sda.out‚ 164.203125# proc.memory.rssMB‚ 2878.37109375* proc.memory.percent‚17.568182945251465 network.recv‚ 1.514758e+06$ disk./.usageGB‚1096.528491973877 disk.dm-1.out‚ 164.203125 disk.rbd3.in‚0 disk.sda2.in‚0 network.sent‚ 1.2023741e+07) disk./.usagePercent‚73.58894714505313$ memory_percent‚98.48754405975342( proc.memory.availableMB‚ 247.80078125 disk.rbd0.out‚0 disk.rbd1.out‚ 2.208984375& disk.nvme0n1.out‚59557.41748046875 disk.rbd2.out‚92.64599609375 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 710.40625# disk.dm-0.out‚59557.41748046875 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.drbd1777.in‚0æ×zq‘£‚(:† óûãÔ—Òµ™ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.598' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9196 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›GrG+¤‚(:  óûãÔŒ¥Õ› disk.sda.out‚ 164.72265625 disk.sda1.in‚0* proc.memory.percent‚17.568182945251465 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 710.40625 disk.dm-1.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚59558.69873046875 disk.dm-1.out‚ 164.72265625 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0# disk.dm-0.out‚59558.69873046875 disk.sda2.in‚0 disk.rbd1.out‚ 2.208984375& disk.nvme0n1.out‚59558.69873046875 disk.nvme0c0n1.in‚ 710.40625 disk.sda1.out‚0 network.recv‚ 1.520233e+06) disk./.usagePercent‚73.58894586501546 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.dm-0.in‚ 710.40625$ memory_percent‚98.48763942718506 disk.rbd3.in‚0( proc.memory.availableMB‚ 247.78515625# proc.memory.rssMB‚ 2878.37109375 cpu‚25.370116660246286 network.sent‚ 1.2131521e+07 disk.sda3.out‚ 164.72265625 disk.drbd1777.out‚0% disk./.usageGB‚1096.5284729003906 disk.rbd2.in‚0 disk.rbd2.out‚92.64599609375]fý2¥- _wandbruntime‚3997  _runtime‚3997?3ënǦÁ & lr-AdamW/pg2‚6.904326730729939e-05 trainer/global_step‚5349 % _timestamp‚1.7884113874618034e+09 _runtime‚3997.873943125 _step‚436 & lr-AdamW/pg1‚6.904326730729939e-05jÊOÙÌ§Æ trainer/global_step‚5349 % _timestamp‚1.7884113874618034e+09 _runtime‚3997.873943125 _step‚436 & lr-AdamW/pg1‚6.904326730729939e-05 & lr-AdamW/pg2‚6.904326730729939e-05´ßü*ƒ2¨- _wandbruntime‚3997  _runtime‚3997¹ùß'©ÙO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7542146956288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚143317076945920 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚150657773361152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7520415296016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7673171781632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70433841 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41113156 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30379918 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167308519720960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178537802 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30379375 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚150657791364096 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚167308756342784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7672959099904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚158200107320320 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚5349 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178517762 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71493074 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7542146956288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70433841 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7542315956224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚158199920317440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚3997.874929824 _step‚437 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108083923 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚150837674402700 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108103961 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚158200107320320 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7542315956224 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159635584561152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7520580417420 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41113154 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚158199920317440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71492529-å¥nN'ªÞO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚150657791364096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚158199920317440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚3997.874929824 _step‚437 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108083923 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚150837674402700 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108103961 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚158200107320320 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7542315956224 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159635584561152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7520580417420 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41113154 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚158199920317440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71492529 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7542146956288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚143317076945920 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚150657773361152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7520415296016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7673171781632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70433841 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41113156 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30379918 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167308519720960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178537802 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30379375 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚150657791364096 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚167308756342784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7672959099904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚158200107320320 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚5349 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178517762 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71493074 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7542146956288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70433841 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7542315956224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178570274 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178550208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚5349 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70446498 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚3998.535847773 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚150685060728832 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7543709107712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7674591736320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71505465 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70446498 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚159664473569280 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7521804460560 _step‚438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108123774 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70446500 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178570274 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚167339089245696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108123774 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7521969582172 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41120533 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚150864824588816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚158228582820352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚143343020128256 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚150685060728832 % _timestamp‚1.7884113881230366e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚150685042725888 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108103710 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30385455 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚150865006749788 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167338852637184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚158228769836544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143343037167616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70446500 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108103710 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178550208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7543540094464 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71505988 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41120531 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚158228769836544 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚150685042725888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30384934 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7543540094464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7543709107712Ç»ýî&­èM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚158228582820352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚143343020128256 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚150685060728832 % _timestamp‚1.7884113881230366e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚150685042725888 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108103710 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30385455 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚150865006749788 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167338852637184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚158228769836544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143343037167616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70446500 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108103710 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178550208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7543540094464 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71505988 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41120531 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚158228769836544 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚150685042725888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30384934 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7543540094464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7543709107712 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178570274 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178550208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚5349 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70446498 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚3998.535847773 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚150685060728832 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7543709107712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7674591736320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71505465 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70446498 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚159664473569280 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7521804460560 _step‚438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108123774 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70446500 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178570274 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚167339089245696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108123774 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7521969582172 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41120533 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚150864824588816¶‡8ú2®- _wandbruntime‚3998  _runtime‚3998É%MX¯Ò /loss/FRt4/assignment_loss‚3.310003546151248 2loss/SRqqt4/assignment_loss‚1.8192154575010826 /loss/FRt1/detection_loss‚3.3282455287196417 /loss/FRt4/detection_loss‚2.4256625710563227 /loss/FBt3/detection_loss‚0.7635355200382529 _runtime‚3998.537445103 /loss/FRt3/assignment_loss‚2.476008482468434 /loss/FBt2/assignment_loss‚0.553373989649117 1loss/FBt3/assignment_loss‚0.16703649998154058 /loss/FRt3/detection_loss‚2.5162449949062786 0loss/SRqqt2/detection_loss‚1.566546962611259 trainer/global_step‚5349 0loss/SRqqt3/detection_loss‚1.581805672943592 epoch‚4 % _timestamp‚1.7884113881253765e+09 2loss/SRqqt1/assignment_loss‚2.1391859763655168 2loss/SRqqt2/assignment_loss‚1.5389009592906824 2loss/SRqqt3/assignment_loss‚1.5862936496734619 1loss/SRqqt4/detection_loss‚1.8657982930027206 /loss/FBt1/detection_loss‚3.0089056864380836 /loss/FRt1/assignment_loss‚4.464304447174072 0loss/SRqqt1/detection_loss‚2.691066854986651 /loss/FBt4/detection_loss‚3.6810718588531017 %loss/total_loss‚49.93579825687113 _step‚439 0loss/FBt4/assignment_loss‚0.3590923620890294 .loss/FBt2/detection_loss‚2.802269846200943 0loss/FRt2/assignment_loss‚2.8023048737010017 0loss/FBt1/assignment_loss‚0.5020210776483509 /loss/FRt2/detection_loss‚1.9869031454207466U"]°× 2loss/SRqqt1/assignment_loss‚2.1391859763655168 2loss/SRqqt2/assignment_loss‚1.5389009592906824 2loss/SRqqt3/assignment_loss‚1.5862936496734619 1loss/SRqqt4/detection_loss‚1.8657982930027206 /loss/FBt1/detection_loss‚3.0089056864380836 /loss/FRt1/assignment_loss‚4.464304447174072 0loss/SRqqt1/detection_loss‚2.691066854986651 /loss/FBt4/detection_loss‚3.6810718588531017 %loss/total_loss‚49.93579825687113 _step‚439 0loss/FBt4/assignment_loss‚0.3590923620890294 .loss/FBt2/detection_loss‚2.802269846200943 0loss/FRt2/assignment_loss‚2.8023048737010017 0loss/FBt1/assignment_loss‚0.5020210776483509 /loss/FRt2/detection_loss‚1.9869031454207466 /loss/FRt4/assignment_loss‚3.310003546151248 2loss/SRqqt4/assignment_loss‚1.8192154575010826 /loss/FRt1/detection_loss‚3.3282455287196417 /loss/FRt4/detection_loss‚2.4256625710563227 /loss/FBt3/detection_loss‚0.7635355200382529 _runtime‚3998.537445103 /loss/FRt3/assignment_loss‚2.476008482468434 /loss/FBt2/assignment_loss‚0.553373989649117 1loss/FBt3/assignment_loss‚0.16703649998154058 /loss/FRt3/detection_loss‚2.5162449949062786 0loss/SRqqt2/detection_loss‚1.566546962611259 trainer/global_step‚5349 0loss/SRqqt3/detection_loss‚1.581805672943592 epoch‚4 % _timestamp‚1.7884113881253765e+09·¥Â›Q›±‚(: ‚üãÔãК gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.828' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.765600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ:9K(²‚(: ‚üãÔ©Âל disk.rbd1.out‚ 2.208984375 disk.dm-0.in‚ 710.40625 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 710.40625 disk.sda3.out‚ 165.14453125& disk.nvme0n1.out‚59559.04638671875 disk.dm-1.out‚ 165.14453125# proc.memory.rssMB‚ 2878.37109375 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda1.out‚0 cpu‚25.368849804015998) disk./.usagePercent‚73.58895354524148 disk.rbd3.out‚0 disk.drbd1777.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.568182945251465 disk.rbd2.in‚0 disk.rbd2.out‚ 92.751953125 disk.sda.in‚0 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 710.40625 disk.dm-1.in‚0$ memory_percent‚98.48790168762207 disk.sda3.in‚0 network.sent‚ 1.2137756e+07 network.recv‚ 1.525141e+06 disk.sda.out‚ 165.14453125 disk.rbd0.in‚0 disk.sda1.in‚0# disk.dm-0.out‚59559.04638671875' proc.memory.availableMB‚ 247.7421875 disk.sda2.out‚0% disk./.usageGB‚1096.5285873413086( disk.nvme0c0n1.out‚59559.04638671875 disk.rbd1.in‚0»«ö<‘³‚(:† ‘üãÔ«â™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.687' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1374 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~µ)´‚(:ž ‘üãÔ¬äæ› disk.rbd2.in‚0 disk.sda.in‚0 disk.sda.out‚ 165.77734375 disk.dm-0.in‚ 710.40625 disk.sda1.in‚0 disk.rbd1.out‚ 2.208984375 network.sent‚ 1.2246242e+07 disk.rbd2.out‚ 92.751953125 disk.rbd3.in‚0( disk.nvme0c0n1.out‚59559.75341796875 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0$ memory_percent‚98.48935604095459( proc.memory.availableMB‚ 247.50390625 disk.nvme0n1.in‚ 710.40625& disk.nvme0n1.out‚59559.75341796875 disk.sda3.out‚ 165.77734375 disk.drbd1777.out‚0 disk.rbd1.in‚0 network.recv‚ 1.531637e+06) disk./.usagePercent‚73.60572843892258% disk./.usageGB‚1096.7785453796387 disk.dm-1.out‚ 165.77734375 disk.sda2.out‚0 disk.sda3.in‚0* proc.memory.percent‚17.568182945251465 disk.drbd1777.in‚0 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 710.40625# disk.dm-0.out‚59559.75341796875 cpu‚25.367678668132633# proc.memory.rssMB‚ 2878.37109375iÉ4O2µ- _wandbruntime‚4032  _runtime‚4032uµä¸Ç¶Á & lr-AdamW/pg1‚6.836289292721019e-05 & lr-AdamW/pg2‚6.836289292721019e-05 trainer/global_step‚5399 % _timestamp‚1.7884114222795255e+09 _runtime‚4032.691654732 _step‚440(zæ!Ì·Æ & lr-AdamW/pg2‚6.836289292721019e-05 trainer/global_step‚5399 % _timestamp‚1.7884114222795255e+09 _runtime‚4032.691654732 _step‚440 & lr-AdamW/pg1‚6.836289292721019e-05¸ë=x12¸- _wandbruntime‚4032  _runtime‚4032úŸæ¥–¹ÙO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109076625 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚152021259600896 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7611818278912 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72133300 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71066789 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚159633246879744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109076625 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7884114222802575e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7589887114424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180143414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7744055035904 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71066789 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚571 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161081231972352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚161081208032256 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚152021241597952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180163454 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30650725 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚168825046192128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159633059876864 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚152021241597952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72132729 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7611987278848 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109096663 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41482006 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7611987278848 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109096663 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚441 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.cuœ Irrent‚569 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7611818278912 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚159633246879744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚144614236062720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71066791 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41482004 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7743838159872 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚152204305337908 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159633059876864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30651294 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚168825287008256 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109096663 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41482006 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7611987278848 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109096663 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚441 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚569 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7611818278912 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚159633246879744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚144614236062720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71066791 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41482004 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7743838159872 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚152204305337908 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159633059876864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30651294 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚168825287008256 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159633059876864 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚152021241597952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72132729 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7611987278848 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0¹OAi2»- _wandbruntime‚4033  _runtime‚4033 ©—è&¼âM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _runtime‚4033.334689166 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180175860 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71079448 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109116476 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7613380430336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚507 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71079448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41489383 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180175860 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7591276278968 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚168855371670016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 $ _timestamp‚1.788411422922038e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109096412 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30656156 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7745469649408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41489381 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚152231455524024 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚168855614569984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180195926 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚159661909395968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚152048510962688 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚509 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7591441400580 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚152048510962688 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7745250689536 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71079450 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109116476 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72145537 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚152231637684996 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159661722379776 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚152048528965632 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180195926 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 trainer/global_step‚5399 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0n”–!½çM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71079450 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 _step‚442 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _runtime‚4033.334689166 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180175860 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71079448 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109116476 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7613380430336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚507 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71079448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41489383 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180175860 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7591276278968 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚168855371670016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 $ _timestamp‚1.788411422922038e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109096412 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30656156 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7745469649408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41489381 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚152231455524024 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚168855614569984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180195926 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚159661909395968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚152048510962688 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚509 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7591441400580 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚152048510962688 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7745250689536 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71079450 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109116476 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72145537 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚152231637684996 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159661722379776 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚152048528965632 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180195926 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 trainer/global_step‚5399 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41850856 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72773310 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚161066199436288 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚4066.853969639 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30922456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110089365 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110069327 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚145911412218880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 trainer/global_step‚5449 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚153384709834752 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚153570936273116 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 181789106 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚162526879383552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚161066199436288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚161066386439168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚161066386439168 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7681489601536 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41850856 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72773310 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚161066199436288 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚4066.853969639 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30922456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110089365 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110069327 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚145911412218880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 trainer/global_step‚5449 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚153384709834752 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚153570936273116 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 181789106 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚162526879383552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚161066199436288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚161066386439168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚161066386439168 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7681489601536 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7683051752960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110089114 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7682882739712 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_en7ÍÊÃd/requested_bytes.large_pool.allocated‚145937355401216 _runtime‚4067.502279596 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚153411979199488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚566 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7816343286784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72786086 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71712398 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41858233 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181801512 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚170372135618560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110109178 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181821578 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181821578 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71712398 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30927855 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚161095048955392 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚153411979199488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚153598268620204 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71712400 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚568 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚170371899010048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚5449 _step‚446 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71712400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚153411997202432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚153598086459232 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7683051752960 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚161094861939200 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110109178 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992ö¶ßí&ÏçM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110089114 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7682882739712 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚145937355401216 _runtime‚4067.502279596 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚153411979199488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚566 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7816343286784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72786086 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71712398 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41858233 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181801512 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚170372135618560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110109178 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181821578 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181821578 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71712398 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30927855 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚161095048955392 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚153411979199488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚153598268620204 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71712400 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚568 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚170371899010048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚5449 _step‚446 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71712400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚153411997202432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚153598086459232 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7683051752960 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚161094861939200 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110109178 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚162555792331776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚162555768391680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7816130618368 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7660913218988 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7683051752960¾‰ÂIÞ2Ð- _wandbruntime‚4067  _runtime‚4067õ…¡ºXÑÒ 0loss/FRt3/assignment_loss‚1.7332902470460305 2loss/SRqqt2/assignment_loss‚1.5022306452904428 2loss/SRqqt3/assignment_loss‚1.5258472430706025 /loss/FRt2/detection_loss‚2.0476220926996005 $ _timestamp‚1.788411457091344e+09 1loss/FBt1/assignment_loss‚0.40117172657280914 _runtime‚4067.503397633 0loss/FRt2/assignment_loss‚2.7856366078732377 2loss/SRqqt1/assignment_loss‚2.4980258129296766 0loss/FBt2/assignment_loss‚1.2714854310266674 0loss/SRqqt1/detection_loss‚2.644626307872034 .loss/FBt4/detection_loss‚2.874758917093277 trainer/global_step‚5449 %loss/total_loss‚49.23887008691388 /loss/FRt4/assignment_loss‚3.792586164040999 2loss/SRqqt4/assignment_loss‚1.7015741945403378 .loss/FRt1/detection_loss‚3.150519148067192 1loss/SRqqt2/detection_loss‚1.5103082165122033 .loss/FBt2/detection_loss‚2.550469660758972 _step‚447 /loss/FRt1/assignment_loss‚4.714950102346915 0loss/FBt4/assignment_loss‚0.2297976549480154 /loss/FRt3/detection_loss‚1.8780680883389254 1loss/SRqqt3/detection_loss‚1.5387280783057213 1loss/SRqqt4/detection_loss‚2.1809248488123822 epoch‚4 0loss/FBt3/assignment_loss‚0.1981453751012623 /loss/FRt4/detection_loss‚2.3086717541922224 /loss/FBt1/detection_loss‚3.1158609059121876 .loss/FBt3/detection_loss‚1.083570863562162&ƒ]Ò× /loss/FRt3/detection_loss‚1.8780680883389254 1loss/SRqqt3/detection_loss‚1.5387280783057213 1loss/SRqqt4/detection_loss‚2.1809248488123822 epoch‚4 0loss/FBt3/assignment_loss‚0.1981453751012623 /loss/FRt4/detection_loss‚2.3086717541922224 /loss/FBt1/detection_loss‚3.1158609059121876 .loss/FBt3/detection_loss‚1.083570863562162 0loss/FRt3/assignment_loss‚1.7332902470460305 2loss/SRqqt2/assignment_loss‚1.5022306452904428 2loss/SRqqt3/assignment_loss‚1.5258472430706025 /loss/FRt2/detection_loss‚2.0476220926996005 $ _timestamp‚1.788411457091344e+09 1loss/FBt1/assignment_loss‚0.40117172657280914 _runtime‚4067.503397633 0loss/FRt2/assignment_loss‚2.7856366078732377 2loss/SRqqt1/assignment_loss‚2.4980258129296766 0loss/FBt2/assignment_loss‚1.2714854310266674 0loss/SRqqt1/detection_loss‚2.644626307872034 .loss/FBt4/detection_loss‚2.874758917093277 trainer/global_step‚5449 %loss/total_loss‚49.23887008691388 /loss/FRt4/assignment_loss‚3.792586164040999 2loss/SRqqt4/assignment_loss‚1.7015741945403378 .loss/FRt1/detection_loss‚3.150519148067192 1loss/SRqqt2/detection_loss‚1.5103082165122033 .loss/FBt2/detection_loss‚2.550469660758972 _step‚447 /loss/FRt1/assignment_loss‚4.714950102346915 0loss/FBt4/assignment_loss‚0.2297976549480154¿$ÿQ œÓ‚(:‘ ÍüãÔûž™ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.225' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.244999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºø§%Ô‚(:š Íüãԛ앜 disk.sda.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 247.3828125( disk./.usagePercent‚73.6117563923238$ disk./.usageGB‚1096.868366241455 disk.dm-0.in‚ 710.40625 disk.dm-1.in‚0# memory_percent‚98.4900951385498* proc.memory.percent‚17.568182945251465 proc.cpu.threads‚17 disk.dm-1.out‚ 167.76953125( disk.nvme0c0n1.out‚59697.88134765625 disk.sda2.out‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0# disk.dm-0.out‚59697.88134765625# proc.memory.rssMB‚ 2878.37109375 disk.sda1.in‚0 disk.nvme0n1.in‚ 710.40625& disk.nvme0n1.out‚59697.88134765625 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 710.40625 disk.drbd1777.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 94.623046875 cpu‚25.362847609524543 network.sent‚ 1.2371956e+07 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd1.out‚ 2.208984375 disk.sda.out‚ 167.76953125 network.recv‚ 1.550304e+06 disk.sda3.out‚ 167.76953125½ƒ›Õ‚(: ÜüãÔñÖÙ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.57' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.913999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C¯;z$Ö‚(:™ ÜüãÔæ™Íœ disk.sda3.out‚ 168.2265625 disk.drbd1777.out‚0% disk.nvme0n1.out‚59831.5439453125 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 710.40625 cpu‚25.361702948836886 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.dm-0.in‚ 710.40625" disk.dm-0.out‚59831.5439453125 disk.dm-1.in‚0 disk.rbd2.out‚94.70947265625* proc.memory.percent‚17.568182945251465% disk./.usageGB‚1096.8678398132324 disk.sda1.in‚0 disk.rbd1.out‚ 2.208984375 disk.dm-1.out‚ 168.2265625 disk.rbd0.in‚0' disk.nvme0c0n1.out‚59831.5439453125' proc.memory.availableMB‚ 247.5546875 network.sent‚ 1.2479575e+07 network.recv‚ 1.556275e+06 disk.drbd1777.in‚0 disk.sda2.in‚0) disk./.usagePercent‚73.61172106328408 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda.out‚ 168.2265625$ memory_percent‚98.48904609680176# proc.memory.rssMB‚ 2878.37109375 disk.rbd2.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17%—î2×- _wandbruntime‚4101  _runtime‚4101é ÆØÀ % _timestamp‚1.7884114912438846e+09 _runtime‚ 4101.65610863 _step‚448 & lr-AdamW/pg1‚6.699112980134676e-05 & lr-AdamW/pg2‚6.699112980134676e-05 trainer/global_step‚5499ëÊrËÙÅ & lr-AdamW/pg1‚6.699112980134676e-05 & lr-AdamW/pg2‚6.699112980134676e-05 trainer/global_step‚5499 % _timestamp‚1.7884114912438846e+09 _runtime‚ 4101.65610863 _step‚448À§c¢a2Ú- _wandbruntime‚4101  _runtime‚4101P’%ÛÙO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 _step‚449 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚171858306010624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73415639 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183394718 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72332691 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7751160924160 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚154748178071552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7728830751240 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7728995872644 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7885779215872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72332689 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚163972502854656 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884114912446368e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚171858079874560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚154937385047560 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111062029 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42219706 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7751160924160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚147208554296320 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚162499525998592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚154748196074496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚162499338995712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚162499525998592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚162499338995712 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚154748178071552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31195419 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202195968 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 trainer/global_step‚5499 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚514 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚163972526794752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72332691 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4101.657045235 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚154748196074496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚516 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73415123 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72332689 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183414758 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111082067 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated-áØFM‚ 111082067 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111062029 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7751329924096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226136064 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7885577019904üuÿºä'ÜÞO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42219704 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚154748178071552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31195419 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202195968 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 trainer/global_step‚5499 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚514 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚163972526794752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72332691 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4101.657045235 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚154748196074496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚516 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73415123 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72332689 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183414758 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111082067 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111082067 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111062029 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7751329924096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226136064 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7885577019904 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 _step‚449 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚171858306010624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73415639 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183394718 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72332691 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7751160924160 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚154748178071552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7728830751240 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7728995872644 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7885779215872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72332689 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚163972502854656 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884114912446368e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚171858079874560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚154937385047560 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111062029 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42219706 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7751160924160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚147208554296320 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚162499525998592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚154748196074496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚162499338995712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚162499525998592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚162499338995712 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183447230 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72345348 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42227083 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚171888643253248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884114918857415e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚164001415802880 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111081816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7752554062336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7730385037396 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73428694 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31201611 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183447230 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚154775447436288 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7887203510272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚154964899555412 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7730219915784 _step‚450 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73428169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42227081 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72345350 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚147234497478656 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚4102.298368289 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72345350 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7752723075584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 183427164 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72345348 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚147234514518016 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111101880O úAî&ßèM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72345350 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚147234497478656 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚4102.298368289 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72345350 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7752723075584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 183427164 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72345348 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚147234514518016 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111101880 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚162528188514816 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚154775465439232 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7752723075584 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183447230 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72345348 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42227083 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚171888643253248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884114918857415e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚164001415802880 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111081816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7752554062336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7730385037396 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73428694 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31201611 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183447230 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚154775447436288 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7887203510272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚154964899555412 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7730219915784 _step‚450 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73428169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42227081 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612Âh(•A2à- _wandbruntime‚4102  _runtime‚4102ÚG†…VáÐ .loss/FRt1/assignment_loss‚5.28493831075471 /loss/FRt3/assignment_loss‚2.218038643951769 0loss/FBt1/assignment_loss‚0.5995844733248911 2loss/SRqqt3/assignment_loss‚1.5683697830067902 0loss/FBt3/assignment_loss‚0.3098220934361085 /loss/FRt2/detection_loss‚2.6277892825215363 1loss/SRqqt1/detection_loss‚2.3952065655163355 .loss/FBt1/detection_loss‚3.167471552888552 /loss/FBt4/detection_loss‚2.6876609325408936 0loss/FRt4/assignment_loss‚3.5888438806647347 2loss/SRqqt2/assignment_loss‚1.7811652402049059 0loss/FBt4/assignment_loss‚0.5538901292685396 % _timestamp‚1.7884114918874023e+09 0loss/SRqqt4/detection_loss‚2.061828647553921 0loss/FRt2/assignment_loss‚2.8812558415324188 1loss/SRqqt1/assignment_loss‚2.626238237108503 1loss/SRqqt4/assignment_loss‚1.888919809833169 1loss/FBt2/assignment_loss‚0.34858338668067923 .loss/FRt4/detection_loss‚2.730051176179023 /loss/FRt3/detection_loss‚1.8012261062308594 _runtime‚4102.299447291 _step‚451 /loss/FRt1/detection_loss‚3.0291212057245187 1loss/SRqqt2/detection_loss‚1.5717592952798491 /loss/SRqqt3/detection_loss‚1.33333488499246 /loss/FBt2/detection_loss‚3.6142240497801037 /loss/FBt3/detection_loss‚0.8105917353370609 %loss/total_loss‚51.47991526431233 epoch‚4 trainer/global_step‚5499 {ÖâÕ 0loss/FRt4/assignment_loss‚3.5888438806647347 2loss/SRqqt2/assignment_loss‚1.7811652402049059 0loss/FBt4/assignment_loss‚0.5538901292685396 % _timestamp‚1.7884114918874023e+09 0loss/SRqqt4/detection_loss‚2.061828647553921 0loss/FRt2/assignment_loss‚2.8812558415324188 1loss/SRqqt1/assignment_loss‚2.626238237108503 1loss/SRqqt4/assignment_loss‚1.888919809833169 1loss/FBt2/assignment_loss‚0.34858338668067923 .loss/FRt4/detection_loss‚2.730051176179023 /loss/FRt3/detect¼E3Uion_loss‚1.8012261062308594 _runtime‚4102.299447291 _step‚451 /loss/FRt1/detection_loss‚3.0291212057245187 1loss/SRqqt2/detection_loss‚1.5717592952798491 /loss/SRqqt3/detection_loss‚1.33333488499246 /loss/FBt2/detection_loss‚3.6142240497801037 /loss/FBt3/detection_loss‚0.8105917353370609 %loss/total_loss‚51.47991526431233 epoch‚4 trainer/global_step‚5499 .loss/FRt1/assignment_loss‚5.28493831075471 /loss/FRt3/assignment_loss‚2.218038643951769 0loss/FBt1/assignment_loss‚0.5995844733248911 2loss/SRqqt3/assignment_loss‚1.5683697830067902 0loss/FBt3/assignment_loss‚0.3098220934361085 /loss/FRt2/detection_loss‚2.6277892825215363 1loss/SRqqt1/detection_loss‚2.3952065655163355 .loss/FBt1/detection_loss‚3.167471552888552 /loss/FBt4/detection_loss‚2.6876609325408936õpù—ã‚(:„ ëüãÔ÷å°™ gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.97' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.594 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉlgD$ä‚(:™ ëüãÔÙªþœ disk.sda.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0 disk.dm-0.in‚ 710.40625 disk.dm-1.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚73.61178455315257% disk./.usageGB‚1096.8687858581543 disk.rbd1.in‚0 disk.nvme0n1.in‚ 710.40625& proc.memory.availableMB‚ 247.734375 disk.sda2.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚59832.4658203125 disk.sda3.out‚ 168.97265625 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.rbd1.out‚ 2.208984375# proc.memory.rssMB‚ 2878.37109375 cpu‚25.3605067719482 disk.rbd2.out‚94.72509765625 disk.nvme0c0n1.in‚ 710.40625 disk.drbd1777.out‚0 disk.rbd3.in‚0" disk.dm-0.out‚59832.4658203125 disk.dm-1.out‚ 168.97265625 disk.rbd0.in‚0% disk.nvme0n1.out‚59832.4658203125 network.recv‚ 1.559669e+06 disk.rbd2.in‚0 disk.sda3.in‚0$ memory_percent‚98.48794937133789 network.sent‚ 1.2485172e+07 proc.cpu.threads‚17 disk.sda.out‚ 168.97265625* proc.memory.percent‚17.568182945251465D5ŠÈœå‚(:‘ úüãÔàž™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.962' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.592399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0° Õæ‚(:Œ úüãÔŠ ëœ disk.sda1.in‚0 proc.memory.rssMB‚2878.375) disk./.usagePercent‚73.61178455315257 disk.rbd2.out‚94.78759765625 disk.sda2.in‚0 disk.sda2.out‚0# disk.dm-0.out‚59832.79736328125 proc.cpu.threads‚17 network.sent‚ 1.2593755e+07 disk.dm-0.in‚ 710.40625 disk.rbd1.out‚ 2.208984375 disk.nvme0n1.in‚ 710.40625 disk.sda.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-1.out‚169.3125& proc.memory.availableMB‚ 247.640625 cpu‚25.35938633411062 network.recv‚ 1.566308e+06( disk.nvme0c0n1.out‚59832.79736328125 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0% disk./.usageGB‚1096.8687858581543 disk.rbd3.out‚0 disk.sda1.out‚0& disk.nvme0n1.out‚59832.79736328125 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚98.48852157592773* proc.memory.percent‚17.568206787109375 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda3.out‚169.3125 disk.sda.out‚169.3125óLB2ç- _wandbruntime‚4136  _runtime‚4136_½'§Ãè½ % lr-AdamW/pg1‚6.63000269289519e-05 % lr-AdamW/pg2‚6.63000269289519e-05 trainer/global_step‚5549 # _timestamp‚1.78841152568264e+09 _runtime‚4136.094790223 _step‚452MÏí¤Èé % lr-AdamW/pg1‚6.63000269289519e-05 % lr-AdamW/pg2‚6.63000269289519e-05 trainer/global_step‚5549 # _timestamp‚1.78841152568264e+09 _runtime‚4136.094790223 _step‚452Äqã<2ê- _wandbruntime‚4136  _runtime‚4136øj‚Þ'ëØO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31469861 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚173374809160704 _step‚453 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 _runtime‚4136.095817216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7821001246720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7820832246784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112054731 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185040410 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74058956 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165418174205952 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7798467691052 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚163932478555136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 $ _timestamp‚1.788411525683472e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7821001246720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7956426467328 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72965639 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚163932665558016 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156111646308352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42588556 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 185020370 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7820832246784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74058415 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚163932478555136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72965639 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚173374576733184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚156304198143532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112074769 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156111664311296 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31470400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7798302569648 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112054731 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156111664311296 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚163932665558016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165418150265856 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚156304015982768 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚156111646308352 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42588554 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257Ù–€Õã'ìÝO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112074769 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156111664311296 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31470400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7798302569648 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112054731 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156111664311296 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚163932665558016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165418150265856 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚156304015982768 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚156111646308352 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42588554 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31469861 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚173374809160704 _step‚453 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 _runtime‚4136.095817216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7821001246720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7820832246784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112054731 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185040410 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74058956 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165418174205952 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7798467691052 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚163932478555136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 $ _timestamp‚1.788411525683472e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7821001246720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7956426467328 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72965639 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚163932665558016 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156111646308352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42588556 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 185020370 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7820832246784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74058415 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚163932478555136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72965639 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚173374576733184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚156304198143532ÅŠ"j&2í- _wandbruntime‚4136  _runtime‚4136}ÆbîãM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156138933676032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚5549 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72978298 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7822225384960 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74071291 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 % _timestamp‚1.7884115263306549e+09 _runtime‚4136.743168177 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185052816 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112074518 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74071810 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31475877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚156331348329648 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72978298 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185072882 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173404902045184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7799856855804 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72978300 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185072882 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚156331530490620 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7822394398208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚163961141058048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42595931 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7822394398208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 185052816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚165447087154176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚163961328074240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42595933 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚156138915673088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _KDevâ-¼aáiceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚165447063214080 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112094582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112074518 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚156138915673088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7822225384960 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7958055693824 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚148531656595456 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7799691734192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156138933676032æÃt†î&ïèM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚148531673634816 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72978300 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31475360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7957838831104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚5549 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72978298 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7822225384960 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74071291 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 % _timestamp‚1.7884115263306549e+09 _runtime‚4136.743168177 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185052816 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112074518 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74071810 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31475877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚156331348329648 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72978298 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185072882 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173404902045184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7799856855804 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72978300 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185072882 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚156331530490620 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7822394398208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚163961141058048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42595931 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7822394398208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 185052816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚165447087154176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚163961328074240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42595933 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚156138915673088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚165447063214080 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112094582 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112074518 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚156138915673088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7822225384960 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7958055693824 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚148531656595456 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7799691734192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156138933676032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156138933676032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113067471 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚157475114545152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73598591 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149802889569280 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7867939509460 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7867774388056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7890672569344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚157670829078740 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113047433 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31743186 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149802872529920 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚165365618114560 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚157475114545152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚157670646917976 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚530 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚174891188842496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚457 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚157475132548096 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚166863821617152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31743714 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8027608041472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8027391165440 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73598589 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚4170.628400476 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186666062 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42957404 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚5599 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚165365805117440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73598589 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚165365618114560 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚166863797677056 % _timestamp‚1.7884115602160344e+09°¸¾„ç þÞO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚157475114545152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚157670646917976 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚530 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚174891188842496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚457 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚157475132548096 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚166863821617152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31743714 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8027608041472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8027391165440 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73598589 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚4170.628400476 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186666062 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42957404 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚5599 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚165365805117440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73598589 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚165365618114560 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚166863797677056 % _timestamp‚1.7884115602160344e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7890503569408 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚174891429658624 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚157475132548096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7890672569344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7890503569408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73598591 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚165365805117440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74700590 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113047433 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74701120 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113067471 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚157475114545152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73598591 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149802889569280 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7867939509460 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7867774388056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7890672569344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚157670829078740 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113047433 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31743186 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149802872529920 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚165365618114560ÉT¡r2ÿ- _wandbruntime‚4171  _runtime‚4171†ò«ïé&€ãM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73611248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42964781 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113067220 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7892065720832 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31749122 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚149828815712256 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 % _timestamp‚1.7884115608535857e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113087284 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42964783 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚157502401912832 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73611250 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149828832751616 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚503 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7891896707584 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113067220 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚165394467633664 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚165394467633664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186698534 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚166892734565376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7892065720832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8029020709888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7869328674212 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7869163552600 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚174921520763904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186698534 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157697979264856 _step‚458 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113087284 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚166892710625280 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8028810138624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186678468 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74713400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74713905 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186678468 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚157502383909888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7891896707584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _runtime‚4171.266336871 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73611250 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416_^2Åî&èM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8029020709888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7869328674212 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7869163552600 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚174921520763904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186698534 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157697979264856 _step‚458 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113087284 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚166892710625280 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8028810138624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186678468 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74713400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74713905 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186678468 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚157502383909888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7891896707584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _runtime‚4171.266336871 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73611250 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73611248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42964781 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113067220 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7892065720832 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31749122 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚149828815712256 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 % _timestamp‚1.7884115608535857e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113087284 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42964783 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚157502401912832 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73611250 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149828832751616 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚503 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7891896707584 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113067220 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚165394467633664 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚165394467633664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186698534 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚166892734565376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7892065720832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944Ê°UL2‚- _wandbruntime‚4171  _runtime‚4171·93YƒÓ 1loss/FBt3/assignment_loss‚0.27421730618696427 0loss/FBt4/assignment_loss‚0.5236445068346386 /loss/FBt2/detection_loss‚4.1537100076675415 .loss/FRt3/detection_loss‚2.475464418912545 /loss/FRt1/assignment_loss‚4.941814581553142 /loss/FBt3/detection_loss‚0.8879128987720551 epoch‚4 .loss/FRt4/detection_loss‚1.963524571978129 0loss/SRqqt1/detection_loss‚2.915061710057435 /loss/FBt4/detection_loss‚1.9616494263921465 _step‚459 &loss/total_loss‚51.326773469058224 0loss/FRt3/assignment_loss‚2.3263105138754234 0loss/FBt1/assignment_loss‚0.6799104628444184 /loss/FRt2/detection_loss‚2.2582916617393494 2loss/SRqqt2/assignment_loss‚1.4025409674819778 2loss/SRqqt3/assignment_loss‚1.4983735723265756 1loss/FBt2/assignment_loss‚0.15883400217959812 0loss/SRqqt2/detection_loss‚1.389884886671515 .loss/FBt1/detection_loss‚3.924250250061353 trainer/global_step‚5599 _runtime‚4171.267751264 % _timestamp‚1.7884115608553941e+09 1loss/SRqqt1/assignment_loss‚2.251942272539492 2loss/SRqqt4/assignment_loss‚1.4970706348121166 /loss/FRt1/detection_loss‚2.9545151432355246 1loss/SRqqt3/detection_loss‚1.5922924935196836 1loss/SRqqt4/detection_loss‚2.1558623760938644 /loss/FRt2/assignment_loss‚3.385467279702425 /loss/FRt4/assignment_loss‚3.754227523620312óçù9^„Ø .loss/FRt4/detection_loss‚1.963524571978129 0loss/SRqqt1/detection_loss‚2.915061710057435 /loss/FBt4/detection_loss‚1.9616494263921465 _step‚459 &loss/total_loss‚51.326773469058224 0loss/FRt3/assignment_loss‚2.3263105138754234 0loss/FBt1/assignment_loss‚0.6799104628444184 /loss/FRt2/detection_loss‚2.2582916617393494 2loss/SRqqt2/assignment_loss‚1.4025409674819778 2loss/SRqqt3/assignment_loss‚1.4983735723265756 1loss/FBt2/assignment_loss‚0.15883400217959812 0loss/SRqqt2/detection_loss‚1.389884886671515 .loss/FBt1/detection_loss‚3.924250250061353 trainer/global_step‚5599 _runtime‚4171.267751264 % _timestamp‚1.7884115608553941e+09 1loss/SRqqt1/assignment_loss‚2.251942272539492 2loss/SRqqt4/assignment_loss‚1.4970706348121166 /loss/FRt1/detection_loss‚2.9545151432355246 1loss/SRqqt3/detection_loss‚1.5922924935196836 1loss/SRqqt4/detection_loss‚2.1558623760938644 /loss/FRt2/assignment_loss‚3.385467279702425 /loss/FRt4/assignment_loss‚3.754227523620312 1loss/FBt3/assignment_loss‚0.27421730618696427 0loss/FBt4/assignment_loss‚0.5236445068346386 /loss/FBt2/detection_loss‚4.1537100076675415 .loss/FRt3/detection_loss‚2.475464418912545 /loss/FRt1/assignment_loss‚4.941814581553142 /loss/FBt3/detection_loss‚0.8879128987720551 epoch‚4Ëm „*‘…‚(:† ¶ýãÔú•™ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6726 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë~I$†‚(:™ ¶ýãÔµÉò› disk.dm-1.in‚0 disk.drbd1777.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd3.out‚0$ memory_percent‚98.49162101745605 disk.drbd1777.out‚0# disk.dm-0.out‚59843.21533203125 proc.memory.rssMB‚2878.375( disk.nvme0c0n1.out‚59843.21533203125 disk.sda1.out‚0 disk.rbd1.in‚0 network.sent‚ 1.2719472e+07 disk.dm-1.out‚ 171.35546875 disk.rbd3.in‚0 disk.sda3.out‚ 171.35546875 disk.rbd0.out‚0 disk.rbd2.in‚0' proc.memory.availableMB‚ 247.1328125* proc.memory.percent‚17.568206787109375 cpu‚25.354928870923807 proc.cpu.threads‚17 disk.rbd1.out‚ 2.208984375 network.recv‚ 1.585444e+06) disk./.usagePercent‚73.62074404882839% disk./.usageGB‚1097.0022888183594 disk.sda.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚59843.21533203125 disk.rbd2.out‚ 96.6669921875 disk.sda.out‚ 171.35546875 disk.sda1.in‚0 disk.nvme0n1.in‚ 710.40625 disk.dm-0.in‚ 710.40625«„‘Z‘‡‚(:† ÅýãÔò“­™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.039' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8078 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zOÉ®~ˆ‚(: ÅýãÔÃËâ› disk.sda1.out‚0 disk.dm-0.in‚ 710.40625 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.in‚0 disk.sda.out‚ 171.91015625 network.sent‚ 1.2827911e+07 disk.rbd1.in‚0# disk.nvme0n1.out‚59943.26953125 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 171.91015625( proc.memory.availableMB‚ 247.23046875 disk.drbd1777.in‚0* proc.memory.percent‚17.568206787109375 proc.cpu.threads‚17 disk.sda1.in‚0# memory_percent‚98.4910249710083 proc.memory.rssMB‚2878.375 network.recv‚ 1.591884e+06) disk./.usagePercent‚73.62070948781125ô;º disk.rbd2.in‚0 disk.dm-0.out‚59943.26953125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.in‚0% disk./.usageGB‚1097.0017738342285 disk.rbd1.out‚ 2.208984375 disk.nvme0n1.in‚ 710.40625 disk.rbd2.out‚ 96.6826171875% disk.nvme0c0n1.out‚59943.26953125 disk.sda2.out‚0 disk.drbd1777.out‚0 cpu‚25.353847755457586 disk.dm-1.out‚ 171.91015625 disk.rbd3.in‚0Ìh‡á2‰- _wandbruntime‚4204  _runtime‚4204¬QžÇŠÁ & lr-AdamW/pg2‚6.490809865923513e-05 trainer/global_step‚5649 % _timestamp‚1.7884115944990268e+09 _runtime‚4204.911159587 _step‚460 & lr-AdamW/pg1‚6.490809865923513e-05,+ÙÌ‹Æ _step‚460 & lr-AdamW/pg1‚6.490809865923513e-05 & lr-AdamW/pg2‚6.490809865923513e-05 trainer/global_step‚5649 % _timestamp‚1.7884115944990268e+09 _runtime‚4204.911159587ÌPñ,>2Œ- _wandbruntime‚4204  _runtime‚4204ÃuÄšß'ÙO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43326256 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚159037277853184 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7937246206464 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32017246 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8098467590144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 trainer/global_step‚5649 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188271674 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114060173 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166798757673984 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚166798944676864 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚158838582781952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74231541 _runtime‚4204.912325944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74231541 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114040135 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75342958 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8098248616960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32016704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚168309469028352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74231539 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43326254 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚176407936618496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 % _timestamp‚1.7884115944998853e+09 _step‚461 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114060173 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚166798944676864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7960174892032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158838600784896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚176407693705216 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74231539 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158838582781952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7960343891968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7960174892032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32017246 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8098467590144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 trainer/global_step‚5649 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188271674 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114060173 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166798757673984 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚166798944676864 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚158838582781952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74231541 _runtime‚4204.912325944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74231541 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114040135 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75342958 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8098248616960 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32016704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚168309469028352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74231539 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43326254 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚176407936618496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 % _timestamp‚1.7884115944998853e+09 _step‚461 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114060173 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚166798944676864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7960174892032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158838600784896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚176407693705216 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74231539 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158838582781952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7960343891968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7960174892032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75355711 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114059922 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7884115951394312e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚158865870149632 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7961737043456 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188324186 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚176438260445696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _step‚462 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114059922 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7938800492620 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8099878469120 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚158865870149632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32022601 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚158865852146688 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚176438025934336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8099667897856 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75356234 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚166827420176896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚159064792361036 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚166827607193088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168338358036480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚166827607193088 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114079986 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74244200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _runtime‚4205.552216908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43333631 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32022080 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚151125991868416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7961737043456 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74244198 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188324186 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188304120 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114079986 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74244200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚168338381976576 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0¿p2ëH‘èM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚166827420176896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚159064792361036 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚166827607193088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168338358036480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚166827607193088 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114079986 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74244200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_ÆÅM;¦!end/requested_bytes.large_pool.peak‚ 4390226944 _runtime‚4205.552216908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43333631 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32022080 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚151125991868416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7961737043456 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74244198 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188324186 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188304120 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114079986 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74244200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚168338381976576 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7961568030208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75355711 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114059922 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7884115951394312e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚158865870149632 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7961737043456 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188324186 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚176438260445696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _step‚462 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114059922 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7938800492620 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8099878469120 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚158865870149632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32022601 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚158865852146688 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚176438025934336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8099667897856 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75356234 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064Î ¥&€2’- _wandbruntime‚4205  _runtime‚4205—U“Ï .loss/FBt2/detection_loss‚3.284708023071289 0loss/SRqqt4/detection_loss‚1.829390119889687 /loss/FBt1/detection_loss‚2.9127592811217675 0loss/FRt4/assignment_loss‚3.1793173786069526 .loss/FRt4/detection_loss‚1.919064661518472 epoch‚4 trainer/global_step‚5649 %loss/total_loss‚50.90626814247095 .loss/FRt1/detection_loss‚3.141592614352703 1loss/SRqqt1/detection_loss‚2.9718046270567795 /loss/FBt4/detection_loss‚2.6303270823425717 2loss/SRqqt4/assignment_loss‚1.7909181372358882 1loss/FBt2/assignment_loss‚0.22518407786680786 0loss/SRqqt3/detection_loss‚1.504602757808955 _runtime‚4205.553523449 _step‚463 .loss/FRt3/detection_loss‚2.371381921853338 % _timestamp‚1.7884115951411636e+09 /loss/FRt2/assignment_loss‚3.314535547134488 1loss/SRqqt1/assignment_loss‚2.664081861232889 0loss/FBt4/assignment_loss‚0.6039952536391158 /loss/FRt3/assignment_loss‚2.061697128840855 2loss/SRqqt3/assignment_loss‚1.9866647802131332 .loss/FRt2/detection_loss‚2.576116848130559 1loss/SRqqt2/detection_loss‚1.5713496126731237 /loss/FBt3/detection_loss‚0.9468917722445398 /loss/FRt1/assignment_loss‚5.185273329416911 2loss/SRqqt2/assignment_loss‚1.5030874525507292 0loss/FBt1/assignment_loss‚0.4691443486556706 0loss/FBt3/assignment_loss‚0.2623795250137223Æ5LQZ”Ô .loss/FRt1/detection_loss‚3.141592614352703 1loss/SRqqt1/detection_loss‚2.9718046270567795 /loss/FBt4/detection_loss‚2.6303270823425717 2loss/SRqqt4/assignment_loss‚1.7909181372358882 1loss/FBt2/assignment_loss‚0.22518407786680786 0loss/SRqqt3/detection_loss‚1.504602757808955 _runtime‚4205.553523449 _step‚463 .loss/FRt3/detection_loss‚2.371381921853338 % _timestamp‚1.7884115951411636e+09 /loss/FRt2/assignment_loss‚3.314535547134488 1loss/SRqqt1/assignment_loss‚2.664081861232889 0loss/FBt4/assignment_loss‚0.6039952536391158 /loss/FRt3/assignment_loss‚2.061697128840855 2loss/SRqqt3/assignment_loss‚1.9866647802131332 .loss/FRt2/detection_loss‚2.576116848130559 1loss/SRqqt2/detection_loss‚1.5713496126731237 /loss/FBt3/detection_loss‚0.9468917722445398 /loss/FRt1/assignment_loss‚5.185273329416911 2loss/SRqqt2/assignment_loss‚1.5030874525507292 0loss/FBt1/assignment_loss‚0.4691443486556706 0loss/FBt3/assignment_loss‚0.2623795250137223 .loss/FBt2/detection_loss‚3.284708023071289 0loss/SRqqt4/detection_loss‚1.829390119889687 /loss/FBt1/detection_loss‚2.9127592811217675 0loss/FRt4/assignment_loss‚3.1793173786069526 .loss/FRt4/detection_loss‚1.919064661518472 epoch‚4 trainer/global_step‚5649 %loss/total_loss‚50.90626814247095ϦGÈ•‚(:… ÔýãÔüÌŒ› gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚98.116' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.6232 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ?Û–‚(:… ÔýãÔ³ô• network.sent‚ 1.2833223e+07 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 710.40625 disk.rbd0.out‚0 disk.rbd2.out‚ 98.3193359375 disk.sda.in‚0 disk.sda.out‚ 172.359375 disk.sda1.out‚0 disk.drbd1777.in‚0" disk.nvme0n1.out‚ 60042.4296875* proc.memory.percent‚17.568206787109375 disk.sda3.in‚0 disk.sda3.out‚ 172.359375 disk.sda1.in‚0 disk.rbd1.out‚ 2.208984375 proc.cpu.threads‚17 network.recv‚ 1.596073e+06 disk.dm-1.in‚0 disk.rbd3.out‚0) disk./.usagePercent‚73.62909834269324% disk./.usageGB‚1097.1267738342285 disk.dm-0.out‚ 60042.4296875 disk.drbd1777.out‚0 disk.dm-1.out‚ 172.359375 disk.rbd2.in‚0$ disk.nvme0c0n1.out‚ 60042.4296875 proc.memory.rssMB‚2878.375 disk.dm-0.in‚ 710.40625 disk.rbd0.in‚0 disk.rbd3.in‚0$ memory_percent‚98.49100112915039& proc.memory.availableMB‚ 247.234375 cpu‚25.352696423641417 X0œ—‚(:‘ ãýãÔçã›™ gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.611' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.322200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡î "$˜‚(:™ ãýãÔ¨¦Ë› disk.nvme0n1.in‚ 710.40625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 98.3349609375 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.out‚0 proc.memory.rssMB‚2878.375 disk.drbd1777.in‚0# disk.dm-0.out‚60142.51318359375 disk.sda3.in‚0( proc.memory.availableMB‚ 246.99609375 network.sent‚ 1.294114e+07 disk.dm-0.in‚ 710.40625 disk.rbd2.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚60142.51318359375 cpu‚25.351648810326598 disk.sda.out‚ 173.16015625 disk.drbd1777.out‚0 disk.rbd1.in‚0* proc.memory.percent‚17.568206787109375 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd0.out‚0 network.recv‚ 1.601692e+06 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 173.16015625% disk./.usageGB‚1097.3760719299316 disk.rbd1.out‚ 2.208984375& disk.nvme0n1.out‚60142.51318359375 disk.sda2.in‚0 disk.sda3.out‚ 173.16015625$ memory_percent‚98.49245548248291) disk./.usagePercent‚73.64582894707091 disk.rbd3.in‚0ëá2™- _wandbruntime‚4239  _runtime‚4239¯r%ÆšÀ & lr-AdamW/pg1‚6.420756333767835e-05 & lr-AdamW/pg2‚6.420756333767835e-05 trainer/global_step‚5699 % _timestamp‚1.7884116290019116e+09 _runtime‚ 4239.41415386 _step‚4647Ô˛Š_runtime‚ 4239.41415386 _step‚464 & lr-AdamW/pg1‚6.420756333767835e-05 & lr-AdamW/pg2‚6.420756333767835e-05 trainer/global_step‚5699 % _timestamp‚1.7884116290019116e+09І@>2œ- _wandbruntime‚4239  _runtime‚4239+3½ß'ÙO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43695106 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 % _timestamp‚1.7884116290026834e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32290130 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚169755116439552 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚169755092499456 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189897326 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168232084236288 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚160202069021696 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8006718024872 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8030015214592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8169112412672 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115052875 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚168232084236288 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115052875 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚160403908788392 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 _runtime‚4239.414996658 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8029846214656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74864491 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74864489 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74864489 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115032837 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚160202051018752 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8030015214592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168231897233408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32289572 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177924439436800 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚160202069021696 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8029846214656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚160202051018752 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚152397207802880 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 trainer/global_step‚5699 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚558 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168231897233408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74864491 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75985236 _step‚465|C/ãžÞO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177924439436800 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚160202069021696 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8029846214656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚160202051018752 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚152397207802880 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 trainer/global_step‚5699 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚558 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168231897233408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74864491 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75985236 _step‚465 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43695106 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 % _timestamp‚1.7884116290026834e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32290130 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚169755116439552 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚169755092499456 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189897326 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168232084236288 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚160202069021696 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8006718024872 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonïZ  itor.on_train_batch_start/segment.all.allocated‚416 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8030015214592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8169112412672 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115052875 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚168232084236288 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115052875 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚160403908788392 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 _runtime‚4239.414996658 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8029846214656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74864491 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74864489 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74864489 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115032837 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚160202051018752 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8030015214592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168231897233408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32289572 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199Ѧ´Ó2Ÿ- _wandbruntime‚4240  _runtime‚4240-­~é& ãM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74877148 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚152423150985216 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 % _timestamp‚1.7884116296429794e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚168260559736320 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚152423133945856 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189929772 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚160431241135272 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74877148 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115072688 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚160431423296244 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 _step‚466 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚168260746752512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8170738620928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _runtime‚4240.055537853 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8031239352832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189929772 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115052624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189949838 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32295656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177954533497344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚168260746752512 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚169784005447680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 trainer/global_step‚5699 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189949838 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74877150 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚160229320383488 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚160229338386432 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8031408366080 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115072688 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74877150 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚160229338386432 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75997596 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32295115 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8031408366080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚541 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8170528049664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8008107189416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43702483Û¼î<î&¡èM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8031408366080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚541 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8170528049664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8008107189416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43702483 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74877148 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚152423150985216 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 % _timestamp‚1.7884116296429794e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚168260559736320 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚152423133945856 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189929772 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚160431241135272 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74877148 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115072688 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚160431423296244 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 _step‚466 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚168260746752512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8170738620928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _runtime‚4240.055537853 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8031239352832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189929772 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115052624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189949838 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32295656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177954533497344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚168260746752512 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚169784005447680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 trainer/global_step‚5699 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189949838 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74877150 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚160229320383488 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚160229338386432 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8031408366080 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115072688 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74877150 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚160229338386432 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75997596 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32295115 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199Òâ"âß2¢- _wandbruntime‚4240  _runtime‚4240c Ç"V£Ð 1loss/FBt1/assignment_loss‚0.25456583083177975 /loss/FRt2/detection_loss‚2.3166880018489304 1loss/SRqqt2/detection_loss‚1.4499048466848976 /loss/FRt1/detection_loss‚2.9730998976477263 /loss/FRt4/assignment_loss‚4.438045649616806 /loss/SRqqt1/detection_loss‚2.67906439942973 .loss/FBt1/detection_loss‚3.021473220416478 .loss/FBt3/detection_loss‚0.887871070028241 /loss/FBt4/detection_loss‚2.0082588081176462 %loss/total_loss‚48.78441604473305 /loss/FRt2/assignment_loss‚3.480947350346765 /loss/FRt3/detection_loss‚1.9845825057400732 .loss/FRt4/detection_loss‚2.113146657193163 trainer/global_step‚5699 _runtime‚ 4240.05666201 _step‚467 0loss/FRt3/assignment_loss‚1.7710381940919526 1loss/SRqqt3/detection_loss‚1.4294926724038446 /loss/FRt1/assignment_loss‚4.762385902733638 0loss/FBt2/assignment_loss‚0.3394361735606383 .loss/FBt2/detection_loss‚3.270640390259879 % _timestamp‚1.7884116296446548e+09 2loss/SRqqt2/assignment_loss‚1.5785835299312192 2loss/SRqqt4/assignment_loss‚1.8649916480536815 0loss/FBt3/assignment_loss‚0.1664044547480729 epoch‚4 1loss/SRqqt1/assignment_loss‚2.330003327557019 2loss/SRqqt3/assignment_loss‚1.4668727322266653 1loss/FBt4/assignment_loss‚0.45792063529538257 1loss/SRqqt4/detection_loss‚1.7389981459688257ß±r¤[¤Õ _step‚467 0loss/FRt3/assignment_loss‚1.7710381940919526 1loss/SRqqt3/detection_loss‚1.4294926724038446 /loss/FRt1/assignment_loss‚4.762385902733638 0loss/FBt2/assignment_loss‚0.3394361735606383 .loss/FBt2/detection_loss‚3.270640390259879 % _timestamp‚1.7884116296446548e+09 2loss/SRqqt2/assignment_loss‚1.5785835299312192 2loss/SRqqt4/assignment_loss‚1.8649916480536815 0loss/FBt3/assignment_loss‚0.1664044547480729 epoch‚4 1loss/SRqqt1/assignment_loss‚2.330003327557019 2loss/SRqqt3/assignment_loss‚1.4668727322266653 1loss/FBt4/assignment_loss‚0.45792063529538257 1loss/SRqqt4/detection_loss‚1.7389981459688257 1loss/FBt1/assignment_loss‚0.25456583083177975 /loss/FRt2/detection_loss‚2.3166880018489304 1loss/SRqqt2/detection_loss‚1.4499048466848976 /loss/FRt1/detection_loss‚2.9730998976477263 /loss/FRt4/assignment_loss‚4.438045649616806 /loss/SRqqt1/detection_loss‚2.67906439942973 .loss/FBt1/detection_loss‚3.021473220416478 .loss/FBt3/detection_loss‚0.887871070028241 /loss/FBt4/detection_loss‚2.0082588081176462 %loss/total_loss‚48.78441604473305 /loss/FRt2/assignment_loss‚3.480947350346765 /loss/FRt3/detection_loss‚1.9845825057400732 .loss/FRt4/detection_loss‚2.113146657193163 trainer/global_step‚5699 _runtime‚ 4240.05666201Óëif¤¥‚(:… òýãÔņž™ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚93.611' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•r$㦂(:” òýãÔÖ’Åœ disk.rbd3.in‚0* proc.memory.percent‚17.568230628967285 disk.rbd0.in‚0# proc.memory.rssMB‚ 2878.37890625 network.sent‚ 1.2947289e+07) disk./.usagePercent‚73.64596207098872 disk.rbd1.in‚0 disk.rbd1.out‚ 2.208984375 disk.rbd0.out‚0 disk.rbd2.out‚98.42138671875( disk.nvme0c0n1.out‚60142.81787109375' proc.memory.availableMB‚ 246.7890625 network.recv‚ 1.60659e+06 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.dm-1.in‚0$ memory_percent‚98.49371910095215 cpu‚25.35057755310579 disk.nvme0c0n1.in‚ 710.40625 proc.cpu.threads‚17 disk.sda.out‚ 173.46875 disk.sda1.in‚0 disk.dm-1.out‚ 173.46875 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚60142.81787109375 disk.rbd3.out‚0% disk./.usageGB‚1097.3780555725098 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 710.40625# disk.dm-0.out‚60142.81787109375 disk.rbd2.in‚0 disk.sda3.out‚ 173.46875 disk.dm-0.in‚ 710.40625@½+ö‘§‚(:† þãÔÂÜ™™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.532' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1064 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð5©ý¨‚(:ò þãԲƲœ disk.sda.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 710.40625 disk.rbd3.in‚0" disk.nvme0c0n1.out‚ 60242.71875$ memory_percent‚98.49112033843994( proc.memory.availableMB‚ 247.21484375 disk.drbd1777.out‚0 disk.rbd2.in‚0* proc.memory.percent‚17.568230628967285 network.recv‚ 1.611993e+06 disk.sda1.out‚0 disk.nvme0n1.in‚ 710.40625 disk.dm-1.in‚0 disk.rbd2.out‚98.43701171875# proc.memory.rssMB‚ 2878.37890625 disk.sda.out‚174 disk.rbd1.out‚ 2.208984375 disk.dm-1.out‚174 disk.nvme0c0n1.in‚ 710.40625) disk./.usagePercent‚73.64595899889831 disk.sda3.out‚174 disk.nvme0n1.out‚ 60242.71875 disk.sda2.in‚0 network.sent‚ 1.3054743e+07 disk.dm-0.out‚ 60242.71875 disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1097.3780097961426 disk.rbd1.in‚0 disk.sda2.out‚0 cpu‚25.34952691976741ØõR2©- _wandbruntime‚4273  _runtime‚4273VþƪÀ & lr-AdamW/pg1‚6.350417500400813e-05 & lr-AdamW/pg2‚6.350417500400813e-05 trainer/global_step‚5749 $ _timestamp‚1.788411663586349e+09 _runtime‚4273.998499708 _step‚468šCSË«Å $ _timestamp‚1.788411663586349e+09 _runtime‚4273.998499708 _step‚468 & lr-AdamW/pg1‚6.350417500400813e-05 & lr-AdamW/pg2‚6.350417500400813e-05 trainer/global_step‚5749ÔDl¤2¬- _wandbruntime‚4273  _runtime‚4273tQ³q­ØO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚169665036792832 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚179440951028224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚169665223795712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚161565537258496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75497441 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8239970301440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QDDeviceStatsMonitor.on_QÓ-Ýmtrain_batch_start/inactive_split.all.allocated‚76626149 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚171200763850752 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚469 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚161565519255552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚179440710212096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75497439 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44063956 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8099686537216 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8099686537216 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8099517537280 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8099517537280 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 $ _timestamp‚1.788411663587117e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116045577 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32562193 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116025539 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32561663 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8076354964684 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚169665223795712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚161770539723600 _runtime‚4273.999487719 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚161565519255552 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8099686537216 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8099517537280 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8099517537280 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 $ _timestamp‚1.788411663587117e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116045577 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32562193 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116025539 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32561663 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8076354964684 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚169665223795712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚161770539723600 _runtime‚4273.999487719 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚161565519255552 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚169665036792832 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚179440951028224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚169665223795712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚161565537258496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75497441 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8239970301440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76626149 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚171200763850752 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚469 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚161565519255552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚179440710212096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75497439 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44063956 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000Õ(¸2¯- _wandbruntime‚4274  _runtime‚4274QÿP™é&°ãM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚169693886311936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚5749 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191575490 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8100910675456 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚153720293062656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116045326 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8241381815296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191575490 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚153720310102016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚169693699295744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚161797872070480 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 % _timestamp‚1.7884116642379544e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚161798054231452 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚533 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76638869 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116045326 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191555424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44071333 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8101079688704 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚161592806623232 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8101079688704 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44071331 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75510098 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚161592806623232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179471034674176 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚169693886311936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚4274.650733697 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191555424 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚161592788620288 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75510100 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75510098 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116065390 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _step‚470 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76638336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75510100 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8100910675456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972»è¸&r±èM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191555424 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚161592788620288 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75510100 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75510098 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116065390 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _step‚470 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76638336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75510100 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8100910675456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚169693886311936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚5749 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191575490 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8100910675456 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚153720293062656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116045326 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8241381815296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191575490 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚153720310102016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚169693699295744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚161797872070480 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 % _timestamp‚1.7884116642379544e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚161798054231452 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚533 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76638869 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116045326 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191555424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44071333 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8101079688704 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚161592806623232 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8101079688704 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44071331 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75510098 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚161592806623232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179471034674176 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚169693886311936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚4274.650733697Öÿî^2²- _wandbruntime‚4274  _runtime‚4274š?a§U³Ï 1loss/SRqqt3/detection_loss‚1.5554785736043144 1loss/SRqqt4/detection_loss‚1.7814819514751434 .loss/FBt2/detection_loss‚3.212698119027274 _step‚471 0loss/FBt1/assignment_loss‚0.4663900898985131 1loss/FBt3/assignment_loss‚0.20251065797997794 /loss/FBt1/detection_loss‚3.2195457599379798 %loss/total_loss‚48.31234457426063 epoch‚4 _runtime‚ 4274.65200943 2loss/SRqqt2/assignment_loss‚1.3621763301222292 1loss/SRqqt3/assignment_loss‚1.362458325508568 /loss/FRt2/detection_loss‚2.3262613008801756 /loss/FRt3/detection_loss‚1.9727508011791441 /loss/FBt3/detection_loss‚0.9122428185462851 1loss/FBt4/assignment_loss‚0.46839115697702466 /loss/FRt3/assignment_loss‚2.203196084499359 0loss/SRqqt4/assignment_loss‚1.78962518470852 1loss/SRqqt2/detection_loss‚1.4376938086870599 /loss/FRt1/assignment_loss‚4.711578726768494 0loss/FBt2/assignment_loss‚0.2399479712921675 trainer/global_step‚5749 .loss/FRt2/assignment_loss‚3.02114639373926 1loss/SRqqt1/assignment_loss‚2.580894198268652 0loss/SRqqt1/detection_loss‚2.587153809145093 % _timestamp‚1.7884116642399487e+09 /loss/FRt1/detection_loss‚2.7737494549330544 /loss/FRt4/detection_loss‚2.3525982931548475 .loss/FBt4/detection_loss‚2.474428199804746 0loss/FRt4/assignment_loss‚3.2979465641227423Ôì‘Z´Ô 0loss/SRqqt4/assignment_loss‚1.78962518470852 1loss/SRqqt2/detection_loss‚1.4376938086870599 /loss/FRt1/assignment_loss‚4.711578726768494 0loss/FBt2/assignment_loss‚0.2399479712921675 trainer/global_step‚5749 .loss/FRt2/assignment_loss‚3.02114639373926 1loss/SRqqt1/assignment_loss‚2.580894198268652 0loss/SRqqt1/detection_loss‚2.587153809145093 % _timestamp‚1.7884116642399487e+09 /loss/FRt1/detection_loss‚2.7737494549330544 /loss/FRt4/detection_loss‚2.3525982931548475 .loss/FBt4/detection_loss‚2.474428199804746 0loss/FRt4/assignment_loss‚3.2979465641227423 1loss/SRqqt3/detection_loss‚1.5554785736043144 1loss/SRqqt4/detection_loss‚1.7814819514751434 .loss/FBt2/detection_loss‚3.212698119027274 _step‚471 0loss/FBt1/assignment_loss‚0.4663900898985131 1loss/FBt3/assignment_loss‚0.20251065797997794 /loss/FBt1/detection_loss‚3.2195457599379798 %loss/total_loss‚48.31234457426063 epoch‚4 _runtime‚ 4274.65200943 2loss/SRqqt2/assignment_loss‚1.3621763301222292 1loss/SRqqt3/assignment_loss‚1.362458325508568 /loss/FRt2/detection_loss‚2.3262613008801756 /loss/FRt3/detection_loss‚1.9727508011791441 /loss/FBt3/detection_loss‚0.9122428185462851 1loss/FBt4/assignment_loss‚0.46839115697702466 /loss/FRt3/assignment_loss‚2.203196084499359×úVû‚‘µ‚(:† þãÔÒ·¡™ gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.226' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6452 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛEö¤¶‚(:Š þãÔ×æü› disk.sda1.in‚0 disk.dm-1.out‚ 174.46875 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda3.out‚ 174.46875# memory_percent‚98.4912633895874( proc.memory.availableMB‚ 247.19140625 disk.drbd1777.out‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 710.40625* proc.memory.percent‚17.568230628967285" disk.nvme0n1.out‚ 60341.8046875 network.sent‚ 1.3060191e+07 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 710.40625 disk.rbd2.out‚98.49951171875 disk.rbd3.out‚0$ disk.nvme0c0n1.out‚ 60341.8046875 disk.sda2.in‚0 cpu‚25.348486934586695% disk./.usageGB‚1097.3780097961426 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 710.40625 disk.dm-0.out‚ 60341.8046875 disk.rbd0.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2878.37890625 network.recv‚ 1.615882e+06) disk./.usagePercent‚73.64595899889831 disk.rbd1.out‚ 2.294921875 proc.cpu.threads‚17 disk.sda.out‚ 174.46875 disk.sda2.out‚0.œ6H·‚(:… ŸþãԉΙ™ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.825' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.565 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷OÁ¸‚(:“ ŸþãÔ̚ƛ disk.nvme0c0n1.in‚ 710.40625& disk.nvme0n1.out‚60342.19580078125 cpu‚25.34745450580266 proc.cpu.threads‚17 network.recv‚ 1.621714e+06 disk.dm-0.in‚ 710.40625 disk.rbd2.in‚0( disk.nvme0c0n1.out‚60342.19580078125 disk.sda.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0& proc.memory.availableMB‚ 247.140625 disk.rbd2.out‚98.51513671875 disk.sda3.in‚0 disk.rbd1.out‚ 2.294921875# proc.memory.rssMB‚ 2878.37890625) disk./.usagePercent‚73.67964984641186# disk.dm-0.out‚60342.19580078125 disk.rbd0.out‚0* proc.memory.percent‚17.568230628967285$ disk./.usageGB‚1097.880027770996 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 175.09375 disk.nvme0n1.in‚ 710.40625$ memory_percent‚98.49157333374023 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 175.09375 disk.sda.out‚ 175.09375 disk.drbd1777.in‚0 network.sent‚ 1.3168193e+07 disk.sda2.in‚0Ø4¤ ‘¹‚(:† ®þãÔ毟™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.703' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9406 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º§ *º‚(:Ÿ ®þãÔôžÀ› disk.sda3.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0! disk.rbd2.out‚100.21435546875 disk.sda.in‚0$ memory_percent‚98.49159717559814 network.recv‚ 1.626698e+06) disk./.usagePercent‚73.67952261066735$ disk./.usageGB‚1097.878131866455 disk.nvme0n1.in‚ 710.40625 disk.dm-0.in‚ 710.40625 disk.drbd1777.in‚0( proc.memory.availableMB‚ 247.13671875 proc.cpu.threads‚17 network.sent‚ 1.3174352e+07 disk.dm-1.out‚ 175.62109375( disk.nvme0c0n1.out‚60441.30029296875& disk.nvme0n1.out‚60441.30029296875 disk.sda2.out‚0 cpu‚25.34636678782226 disk.rbd1.in‚0 disk.rbd1.out‚ 2.357421875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2878.37890625# disk.dm-0.out‚60441.30029296875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda3.out‚ 175.62109375 disk.sda.out‚ 175.62109375 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.dm-1.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.568230628967285Z-š2»- _wandbruntime‚4308  _runtime‚4308—]™Ç¼Á & lr-AdamW/pg2‚6.279808024329895e-05 trainer/global_step‚5799 % _timestamp‚1.7884116981470995e+09 _runtime‚4308.559241557 _step‚472 & lr-AdamW/pg1‚6.279808024329895e-05XzÌ½Æ _runtime‚4308.559241557 _step‚472 & lr-AdamW/pg1‚6.279808024329895e-05 & lr-AdamW/pg2‚6.279808024329895e-05 trainer/global_step‚5799 % _timestamp‚1.7884116981470995e+09ØÆ´6E2¾- _wandbruntime‚4308  _runtime‚4308|W´ß'¿ÙO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8169188859904 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117038279 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76130391 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 _runtime‚4308.560170978 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193148630 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚171098176352256 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚163137352819572 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8169357859840 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚171098363355136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8169188859904 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 % _timestamp‚1.7884116981478195e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚180957475057664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32835304 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172646411261952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117018241 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8169357859840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77268657 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚180957240532992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77268108 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162928987492352 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚171098176352256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚549 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚154991508997120 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8145826783092 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8311063795712 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76130389 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162929005495296 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚171098363355136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162929005495296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚473 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ,R'ä'ÀÞO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162929005495296 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚171098363355136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162929005495296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚473 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8169188859904 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117038279 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76130391 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 _runtime‚4308.560170978 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193148630 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚171098176352256 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚163137352819572 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8169357859840 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚171098363355136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8169188859904 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 % _timestamp‚1.7884116981478195e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚180957475057664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32835304 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172646411261952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117018241 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8169357859840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77268657 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚180957240532992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77268108 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162928987492352 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚171098176352256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚549 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚154991508997120 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8145826783092 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8311063795712 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76130389 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416Ù ‘2Á- _wandbruntime‚4309  _runtime‚4309%Èà‰*ÂãM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193181076 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76143050 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8170581998080 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44440181 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32840627 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1711268388551²Žµ¿#68 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162956256857088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8170581998080 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚180987574481408 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚155017469218816 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193181076 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76143048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚171127025871360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117058092 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193201142 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚163164503005688 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚172675300270080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162956274860032 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117058092 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77281349 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚172675324210176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8170751011328 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚180987806895616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77280808 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚474 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193201142 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8170751011328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚4309.205082205 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76143048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8147215947844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117038028 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162956256857088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8312274211328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1ÚÞ«î&ÃèM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚171127025871360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117058092 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193201142 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚163164503005688 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚172675300270080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162956274860032 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117058092 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77281349 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚172675324210176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8170751011328 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚180987806895616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77280808 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚474 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193201142 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8170751011328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚4309.205082205 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76143048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8147215947844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117038028 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162956256857088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8312274211328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193181076 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76143050 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8170581998080 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44440181 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32840627 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚171126838855168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162956256857088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8170581998080 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚180987574481408 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚155017469218816 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193181076 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76143048Úž“­N2Ä- _wandbruntime‚4309  _runtime‚4309ºXrXÅÒ 0loss/FBt4/assignment_loss‚0.8443032510959628 /loss/FRt1/detection_loss‚2.9723480302712013 0loss/SRqqt1/detection_loss‚2.493389267345955 1loss/SRqqt2/detection_loss‚1.7415375895187504 /loss/FBt3/detection_loss‚0.7805594838587795 0loss/FBt2/assignment_loss‚0.3675798440781364 /loss/FRt3/detection_loss‚1.8534287655470418 1loss/SRqqt4/detection_loss‚1.6842742607755177 /loss/FRt3/assignment_loss‚2.099804470352098 .loss/FBt4/detection_loss‚2.431159095330672 trainer/global_step‚5799 2loss/SRqqt4/assignment_loss‚1.9228821432186385 0loss/FBt1/assignment_loss‚0.4134157273778588 .loss/FRt2/detection_loss‚2.521345690602348 _runtime‚4309.206107627 0loss/FRt1/assignment_loss‚5.3401534721769135 0loss/FRt2/assignment_loss‚2.8876836583727883 .loss/FRt4/detection_loss‚2.298188834057914 epoch‚4 1loss/SRqqt3/detection_loss‚1.6604965848893654 0loss/FRt4/assignment_loss‚3.6724117822117277 2loss/SRqqt1/assignment_loss‚2.7828477456651886 2loss/SRqqt2/assignment_loss‚1.4132306228895657 1loss/FBt3/assignment_loss‚0.23185077158674347 /loss/FBt2/detection_loss‚3.4216445088386536 %loss/total_loss‚49.97433933434856 $ _timestamp‚1.788411698794044e+09 1loss/SRqqt3/assignment_loss‚1.455028730921629 .loss/FBt1/detection_loss‚2.684775003365108 _step‚475Ô¢r ]Æ× 0loss/FBt4/assignment_loss‚0.8443032510959628 /loss/FRt1/detection_loss‚2.9723480302712013 0loss/SRqqt1/detection_loss‚2.493389267345955 1loss/SRqqt2/detection_loss‚1.7415375895187504 /loss/FBt3/detection_loss‚0.7805594838587795 0loss/FBt2/assignment_loss‚0.3675798440781364 /loss/FRt3/detection_loss‚1.8534287655470418 1loss/SRqqt4/detection_loss‚1.6842742607755177 /loss/FRt3/assignment_loss‚2.099804470352098 .loss/FBt4/detection_loss‚2.431159095330672 trainer/global_step‚5799 2loss/SRqqt4/assignment_loss‚1.9228821432186385 0loss/FBt1/assignment_loss‚0.4134157273778588 .loss/FRt2/detection_loss‚2.521345690602348 _runtime‚4309.206107627 0loss/FRt1/assignment_loss‚5.3401534721769135 0loss/FRt2/assignment_loss‚2.8876836583727883 .loss/FRt4/detection_loss‚2.298188834057914 epoch‚4 1loss/SRqqt3/detection_loss‚1.6604965848893654 0loss/FRt4/assignment_loss‚3.6724117822117277 2loss/SRqqt1/assignment_loss‚2.7828477456651886 2loss/SRqqt2/assignment_loss‚1.4132306228895657 1loss/FBt3/assignment_loss‚0.23185077158674347 /loss/FBt2/detection_loss‚3.4216445088386536 %loss/total_loss‚49.97433933434856 $ _timestamp‚1.788411698794044e+09 1loss/SRqqt3/assignment_loss‚1.455028730921629 .loss/FBt1/detection_loss‚2.684775003365108 _step‚475Û7û­Ç‚(:„ ½þãÔýÁ¯™ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.115' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.623 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KW‘#È‚(:˜ ½þãÔ€èþ› disk.sda1.out‚0 disk.drbd1777.in‚0& disk.nvme0n1.out‚60540.74560546875 disk.rbd3.out‚0* proc.memory.percent‚17.568230628967285% disk./.usageGB‚1097.8783149719238 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 710.40625 disk.dm-0.in‚ 710.40625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0$ memory_percent‚98.49181175231934' disk./.usagePercent‚73.679534899029 disk.sda.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 network.recv‚ 1.631095e+06 disk.rbd3.in‚0 disk.sda3.out‚ 176.453125 proc.cpu.threads‚17 network.sent‚ 1.3180459e+07 disk.rbd1.out‚ 2.357421875 disk.dm-1.out‚ 176.453125( disk.nvme0c0n1.out‚60540.74560546875' proc.memory.availableMB‚ 247.1015625 disk.sda.out‚ 176.453125# disk.dm-0.out‚60540.74560546875 disk.rbd0.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2878.37890625 disk.rbd1.in‚0 disk.rbd2.in‚0! disk.rbd2.out‚100.22607421875 disk.nvme0c0n1.in‚ 710.40625 cpu‚25.345337810164047kᑛɂ(: ÌþãÔòÒµ™ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.648' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.729599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚Ç´Ê‚(: ÌþãÔùÅÛ› disk.sda1.out‚0 disk.drbd1777.in‚0# proc.memory.rssMB‚ 2878.37890625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚100.3125 disk.rbd3.out‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚60541.2822265625 network.sent‚ 1.328814e+07 disk.sda2.out‚0 disk.drbd1777.out‚0 disk.dm-1.in‚0$ memory_percent‚98.49328994750977 cpu‚25.344324827184938 disk.sda.out‚ 176.953125* proc.memory.percent‚17.568230628967285 network.recv‚ 1.636981e+06' disk.nvme0c0n1.out‚60541.2822265625 disk.dm-1.out‚ 176.953125 disk.rbd2.in‚0 disk.rbd1.out‚ 2.3896484375 disk.nvme0n1.in‚ 710.40625" disk.dm-0.out‚60541.2822265625% disk./.usageGB‚1097.8782005310059 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda3.out‚ 176.953125 disk.dm-0.in‚ 710.40625& proc.memory.availableMB‚ 246.859375 proc.cpu.threads‚17) disk./.usagePercent‚73.67952721880296 disk.sda2.in‚0 disk.sda.in‚0ä·u42Ë- _wandbruntime‚4343  _runtime‚4343C`~Ä̾ _step‚476 % lr-AdamW/pg1‚6.20894262046406e-05 % lr-AdamW/pg2‚6.20894262046406e-05 trainer/global_step‚5849 % _timestamp‚1.7884117327606182e+09 _runtime‚ 4343.17275005Èy“ÝÉÍà % lr-AdamW/pg1‚6.20894262046406e-05 % lr-AdamW/pg2‚6.20894262046406e-05 trainer/global_step‚5849 % _timestamp‚1.7884117327606182e+09 _runtime‚ 4343.17275005 _step‚476Üx.Þë2Î- _wandbruntime‚4343  _runtime‚4343·9o£²ÏØO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚164292473732096 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚164292455729152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33107987 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172531315911680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8239029182464 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77910178 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8215133480096 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 194774282 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118010943 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44801654 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚156288685153280 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚156288668113920 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118010943 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _step‚477 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77909641 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚164503801594016 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76763339 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8381978257408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚164503983754780 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMÆÚJI, onitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 $ _timestamp‚1.788411732761394e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚164292455729152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚537 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76763339 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8239029182464 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76763341 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚535 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8381769769984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚4343.173740636 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8238860182528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172531315911680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8239029182464 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77910178 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8215133480096 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 194774282 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118010943 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44801654 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚156288685153280 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚156288668113920 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118010943 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _step‚477 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77909641 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚164503801594016 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76763339 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8381978257408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚164503983754780 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 $ _timestamp‚1.788411732761394e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚164292455729152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚537 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76763339 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8239029182464 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76763341 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚535 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8381769769984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚4343.173740636 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8238860182528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _runtime‚4343.814574417 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33114000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118050794 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77922494 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚156314611296256 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚164531316101868 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8216687766252 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚172559978414592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚172560165430784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76776000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118030730 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194826794 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 194806728 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8383397446144 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚164319743096832 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _step‚478 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚172560165430784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118050794 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚156314628335616 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76775998 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76775998 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚164531133940896 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 194806728 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8240253320704 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 % _timestamp‚1.7884117334019573e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77923033 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 trainer/global_step‚5849 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76776000 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8240422333952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8216522644640 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416žæ£³%ÓèM _step‚478 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚172560165430784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118050794 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚156314628335616 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76775998 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76775998 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚164531133940896 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 194806728 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8240253320704 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 % _timestamp‚1.7884117334019573e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77923033 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 trainer/global_step‚5849 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76776000 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8240422333952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8216522644640 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚164319725093888 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚164319725093888 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _runtime‚4343.814574417 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33114000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118050794 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77922494 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚156314611296256 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚164531316101868 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8216687766252 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚172559978414592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚172560165430784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76776000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118030730 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194826794 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 194806728 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8383397446144 FDeváý;iceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚164319743096832 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199Þ|ÇG2Ô- _wandbruntime‚4343  _runtime‚4343e`‚'WÕÑ 0loss/FBt1/assignment_loss‚0.4531503223510559 .loss/FRt1/detection_loss‚3.088973309682763 /loss/FRt3/detection_loss‚2.0870193873665164 .loss/FRt4/detection_loss‚2.074879341303034 %loss/total_loss‚50.30482708586388 $ _timestamp‚1.788411733403586e+09 /loss/FBt3/assignment_loss‚0.269754857229493 0loss/FRt1/assignment_loss‚4.5369525577711025 /loss/FRt3/assignment_loss‚2.708277509680816 1loss/SRqqt1/detection_loss‚2.8765511214733124 1loss/SRqqt2/detection_loss‚1.4663779546072087 0loss/FBt4/assignment_loss‚0.7119395290777675 2loss/SRqqt3/assignment_loss‚1.5021207966964762 .loss/FBt4/detection_loss‚2.219983134418726 epoch‚4 1loss/SRqqt3/detection_loss‚1.4975094264900346 1loss/SRqqt4/detection_loss‚1.8774157136678695 trainer/global_step‚5849 /loss/FRt2/assignment_loss‚2.722346529364586 2loss/SRqqt1/assignment_loss‚2.6021015216474948 0loss/FBt2/assignment_loss‚0.4548874981701374 _step‚479 0loss/FRt4/assignment_loss‚3.5712247047018497 /loss/FBt1/detection_loss‚3.3894830516406467 _runtime‚4343.815959125 /loss/FRt2/detection_loss‚2.3081393599510194 2loss/SRqqt2/assignment_loss‚1.4992678062121074 2loss/SRqqt4/assignment_loss‚2.1128789928555487 .loss/FBt2/detection_loss‚3.248088444982256 /loss/FBt3/detection_loss‚1.0255042145220572»ZB¹\ÖÖ $ _timestamp‚1.788411733403586e+09 /loss/FBt3/assignment_loss‚0.269754857229493 0loss/FRt1/assignment_loss‚4.5369525577711025 /loss/FRt3/assignment_loss‚2.708277509680816 1loss/SRqqt1/detection_loss‚2.8765511214733124 1loss/SRqqt2/detection_loss‚1.4663779546072087 0loss/FBt4/assignment_loss‚0.7119395290777675 2loss/SRqqt3/assignment_loss‚1.5021207966964762 .loss/FBt4/detection_loss‚2.219983134418726 epoch‚4 1loss/SRqqt3/detection_loss‚1.4975094264900346 1loss/SRqqt4/detection_loss‚1.8774157136678695 trainer/global_step‚5849 /loss/FRt2/assignment_loss‚2.722346529364586 2loss/SRqqt1/assignment_loss‚2.6021015216474948 0loss/FBt2/assignment_loss‚0.4548874981701374 _step‚479 0loss/FRt4/assignment_loss‚3.5712247047018497 /loss/FBt1/detection_loss‚3.3894830516406467 _runtime‚4343.815959125 /loss/FRt2/detection_loss‚2.3081393599510194 2loss/SRqqt2/assignment_loss‚1.4992678062121074 2loss/SRqqt4/assignment_loss‚2.1128789928555487 .loss/FBt2/detection_loss‚3.248088444982256 /loss/FBt3/detection_loss‚1.0255042145220572 0loss/FBt1/assignment_loss‚0.4531503223510559 .loss/FRt1/detection_loss‚3.088973309682763 /loss/FRt3/detection_loss‚2.0870193873665164 .loss/FRt4/detection_loss‚2.074879341303034 %loss/total_loss‚50.30482708586388ßmWž›×‚(:… ÛþãÔö‘™ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚89.186' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.8372 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EC’Ý#Ø‚(:˜ ÛþãÔÁ¨¶› disk.rbd2.out‚ 100.328125 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1777.in‚0 cpu‚25.34337627034373 proc.cpu.threads‚17" disk.dm-0.out‚60640.8212890625 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd1777.out‚0% disk.nvme0n1.out‚60640.8212890625$ memory_percent‚98.49190711975098* proc.memory.percent‚17.568230628967285 network.sent‚ 1.3293655e+07 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda3.out‚ 177.71484375 disk.sda.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda.out‚ 177.71484375 disk.rbd1.in‚0 disk.rbd1.out‚ 2.3896484375# proc.memory.rssMB‚ 2878.37890625 network.recv‚ 1.640743e+06' disk.nvme0c0n1.out‚60640.8212890625 disk.dm-0.in‚ 710.40625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0) disk./.usagePercent‚73.67952721880296 disk.dm-1.out‚ 177.71484375 disk.sda1.in‚0 disk.nvme0n1.in‚ 710.40625% disk./.usageGB‚1097.8782005310059' proc.memory.availableMB‚ 247.0859375\¬>Ù‚(:… êþãÔ÷ºº™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.115' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.623 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]h÷—Ú‚(:“ êþãԱƜ disk.rbd2.out‚ 100.390625 disk.rbd3.out‚0 disk.drbd1777.in‚0 disk.rbd1.out‚ 2.45263671875 disk.dm-0.in‚ 710.40625$ memory_percent‚98.49374294281006* proc.memory.percent‚17.568230628967285 disk.dm-1.out‚ 178.07421875 cpu‚25.342309722414587 network.sent‚ 1.3402291e+07 disk.rbd0.out‚0 disk.rbd3.in‚0& disk.nvme0c0n1.out‚60740.029296875 disk.sda3.out‚ 178.07421875 disk.sda.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0$ disk./.usageGB‚1097.877815246582 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 710.40625# proc.memory.rssMB‚ 2878.37890625 disk.dm-1.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 246.78515625 proc.cpu.threads‚17$ disk.nvme0n1.out‚60740.029296875' disk./.usagePercent‚73.679501362042 disk.sda2.out‚0 disk.sda.out‚ 178.07421875 disk.drbd1777.out‚0! disk.dm-0.out‚60740.029296875 network.recv‚ 1.6472e+06ù•yµ2Û- _wandbruntime‚4377  _runtime‚4377qtw_ÇÜÁ & lr-AdamW/pg1‚6.137836057047258e-05 & lr-AdamW/pg2‚6.137836057047258e-05 trainer/global_step‚5899 % _timestamp‚1.7884117671452854e+09 _runtime‚4377.557394749 _step‚480`RhÌÝÆ % _timestamp‚1.7884117671452854e+09 _runtime‚4377.557394749 _step‚480 & lr-AdamW/pg1‚6.137836057047258e-05 & lr-AdamW/pg2‚6.137836057047258e-05 trainer/global_step‚5899àe Òj2Þ- _wandbruntime‚4377  _runtime‚43774-Z·Þ'ßØO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚165870614689988 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚175537706084352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8308700505088 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚183990507780608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119023683 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77396291 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45170506 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119023683 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚173964642473984 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚165655923965952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚165655941968896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45170504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77396291 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 trainer/global_step‚5899 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚165655941968896 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157585827230720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚4377.558342263 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78551540 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78550990 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚165870432529224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8284770419908 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8284605298504 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8308531505152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77396289 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33381034 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚173964455471104 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚157585844270080,>ñ2ã'àÝO JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 trainer/global_step‚5899 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚165655941968896 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157585827230720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚4377.558342263 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78551540 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78550990 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚165870432529224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8284770419908 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8284605298504 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8308531505152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77396289 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33381034 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚173964455471104 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚157585844270080 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚165870614689988 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚175537706084352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8308700505088 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚183990507780608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119023683 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77396291 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45170506 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119023683 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚173964642473984 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚165655923965952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚165655941968896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45170504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77396291 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936áö¯ÎO2á- _wandbruntime‚4378  _runtime‚4378 \ê%âãM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184020837427200 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚165897764876104 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119043496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚165683193330688 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚157611770413056 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚5899 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77408950 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8310093656576 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45177883 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77408948 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33385838 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚165897947037076 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196452446 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78563719 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8454218394624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚173993117974016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196432380 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚533 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 % _timestamp‚1.7884117678028579e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚165683211333632 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚157611787452416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚165683193330688 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚173993304990208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8285994463048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45177881 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚173993304990208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8309924643328 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚175566595092480 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0.y[î&ãèM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚175566595092480 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184020837427200 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚165897764876104 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119043496 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚165683193330688 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚157611770413056 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚5899 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77408950 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8310093656576 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45177883 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77408948 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33385838 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚165897947037076 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196452446 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78563719 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8454218394624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚173993117974016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196432380 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚533 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 % _timestamp‚1.7884117678028579e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚165683211333632 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚157611787452416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚165683193330688 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚173993304990208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8285994463048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45177881 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚173993304990208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8309924643328 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217âj6e2ä- _wandbruntime‚4378  _runtime‚4378éñ‹SåÍ /loss/FRt3/detection_loss‚2.3316323020391994 1loss/SRqqt4/detection_loss‚1.9399210640857385 0loss/FBt3/assignment_loss‚0.2456960066766164 -loss/FRt4/detection_loss‚2.22167822787928 0loss/SRqqt3/detection_loss‚1.446059469619523 /loss/FBt3/detection_loss‚0.7783654907283668 % _timestamp‚1.7884117678050864e+09 2loss/SRqqt4/assignment_loss‚1.3966593447812767 _runtime‚4378.217149904 _step‚483 .loss/FRt2/detection_loss‚2.146005840735002 1loss/SRqqt2/detection_loss‚1.3591998843941837 /loss/FBt2/detection_loss‚2.9014765545725822 &loss/total_loss‚48.140548958909505 1loss/SRqqt1/detection_loss‚2.8719533152050443 epoch‚4 1loss/SRqqt1/assignment_loss‚2.604922663282465 0loss/FBt2/assignment_loss‚0.3724810682426778 /loss/FBt1/detection_loss‚3.0824375331401823 .loss/FBt4/detection_loss‚2.348849790436881 .loss/FRt1/assignment_loss‚4.82725579398019 /loss/FRt4/assignment_loss‚3.137010402457659 1loss/SRqqt2/assignment_loss‚1.513106764946133 trainer/global_step‚5899 /loss/FRt2/assignment_loss‚3.034710257703608 0loss/FBt4/assignment_loss‚0.5421761827679542 0loss/FRt3/assignment_loss‚2.3169230454497867 1loss/SRqqt3/assignment_loss‚1.169661719513976 0loss/FBt1/assignment_loss‚0.4322186674322438 .loss/FRt1/detection_loss‚3.120147568838937h¨<XæÒ .loss/FRt2/detection_loss‚2.146005840735002 1loss/SRqqt2/detection_loss‚1.3591998843941837 /loss/FBt2/detection_loss‚2.9014765545725822 &loss/total_loss‚48.140548958909505 1loss/SRqqt1/detection_loss‚2.8719533152050443 epoch‚4 1loss/SRqqt1/assignment_loss‚2.604922663282465 0loss/FBt2/assignment_loss‚0.3724810682426778 /loss/FBt1/detection_loss‚3.0824375331401823 .loss/FBt4/detection_loss‚2.348849790436881 .loss/FRt1/assignment_loss‚4.82725579398019 /loss/FRt4/assignment_loss‚3.137010402457659 1loss/SRqqt2/assignment_loss‚1.513106764946133 trainer/global_step‚5899 /loss/FRt2/assignment_loss‚3.034710257703608 0loss/FBt4/assignment_loss‚0.5421761827679542 0loss/FRt3/assignment_loss‚2.3169230454497867 1loss/SRqqt3/assignment_loss‚1.169661719513976 0loss/FBt1/assignment_loss‚0.4322186674322438 .loss/FRt1/detection_loss‚3.120147568838937 /loss/FRt3/detection_loss‚2.3316323020391994 1loss/SRqqt4/detection_loss‚1.9399210640857385 0loss/FBt3/assignment_loss‚0.2456960066766164 -loss/FRt4/detection_loss‚2.22167822787928 0loss/SRqqt3/detection_loss‚1.446059469619523 /loss/FBt3/detection_loss‚0.7783654907283668 % _timestamp‚1.7884117678050864e+09 2loss/SRqqt4/assignment_loss‚1.3966593447812767 _runtime‚4378.217149904 _step‚483ã8îØq‘ç‚(:† ùþãÔî£Æ™ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.423' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0846 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!i K(è‚(: ùþãÔ ÷“œ( disk.nvme0c0n1.out‚60839.71826171875 disk.sda2.out‚0 network.recv‚ 1.651462e+06 disk.drbd1777.in‚0 cpu‚25.34125575544396 network.sent‚ 1.3408362e+07 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 246.7890625 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda.out‚ 178.62890625 disk.rbd1.out‚ 2.45263671875 disk.sda3.out‚ 178.62890625# proc.memory.rssMB‚ 2878.37890625 disk.nvme0n1.in‚ 710.40625 disk.sda3.in‚0* proc.memory.percent‚17.568230628967285$ disk./.usageGB‚1097.876262664795 disk.dm-0.in‚ 710.40625# disk.dm-0.out‚60839.71826171875 disk.rbd0.in‚0 disk.rbd2.out‚ 101.98046875 disk.rbd3.out‚0$ memory_percent‚98.49371910095215) disk./.usagePercent‚73.67939716697556 disk.sda1.in‚0 disk.dm-1.out‚ 178.62890625 disk.rbd2.in‚0& disk.nvme0n1.out‚60839.71826171875 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 710.40625 disk.rbd1.in‚0Á_}Ýé‚(:… ˆÿãÔ¤«ª™ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚91.102' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.2204 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ »%ê‚(:š ˆÿãÔ¯‡«œ disk.rbd2.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.sda3.out‚ 179.13671875 disk.drbd1777.out‚0* proc.memory.percent‚17.568230628967285 disk.rbd0.out‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 246.85546875 cpu‚25.34026510406832 disk.rbd2.out‚ 102.04296875 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 1.3516222e+07% disk./.usageGB‚1097.8762702941895 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 710.40625 network.recv‚ 1.657603e+06# proc.memory.rssMB‚ 2878.37890625( disk.nvme0c0n1.out‚60840.02294921875 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda.out‚ 179.13671875 disk.rbd1.out‚2.515625 disk.dm-0.in‚ 710.40625 disk.dm-1.out‚ 179.13671875 disk.nvme0c0n1.in‚ 710.40625 disk.sda.in‚0& disk.nvme0n1.out‚60840.02294921875$ memory_percent‚98.49331378936768 disk.rbd0.in‚0) disk./.usagePercent‚73.67939767899063 disk.dm-1.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0# disk.dm-0.out‚60840.02294921875†29œë‚(:‘ —ÿãÔ„Ù©™ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.363' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.672600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚0S"ì‚(:Ž —ÿãÔÓÇžœ disk.rbd1.in‚0 network.sent‚ 1.352169e+07% disk./.usageGB‚1097.8762512207031 disk.sda1.out‚0" disk.dm-0.out‚60939.1396484375 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.rbd1.out‚2.515625% disk.nvme0n1.out‚60939.1396484375 disk.sda3.out‚ 179.71875* proc.memory.percent‚17.568230628967285 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0$ memory_percent‚98.49488735198975# proc.memory.rssMB‚ 2878.37890625 disk.dm-1.out‚ 179.71875 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 710.40625 cpu‚25.339340835084002 disk.sda2.in‚0 disk.nvme0n1.in‚ 710.40625 disk.sda.in‚0 disk.sda.out‚ 179.71875 disk.dm-1.in‚0 disk.rbd2.out‚ 102.05859375 disk.dm-0.in‚ 710.40625 disk.rbd3.in‚0' disk.nvme0c0n1.out‚60939.1396484375 proc.cpu.threads‚17 network.recv‚ 1.661062e+06 disk.sda3.in‚0( proc.memory.availableMB‚ 246.59765625) disk./.usagePercent‚73.67939639895297·ëÉ2í- _wandbruntime‚4412  _runtime‚4412.¯}ÇîÁ % _timestamp‚1.7884118020064707e+09 _runtime‚ 4412.4186037 _step‚484 ' lr-AdamW/pg1‚6.0665031525807304e-05 ' lr-AdamW/pg2‚6.0665031525807304e-05 trainer/global_step‚5949xbíòÌïÆ ' lr-AdamW/pg2‚6.0665031525807304e-05 trainer/global_step‚5949 % _timestamp‚1.7884118020064707e+09 _runtime‚ 4412.4186037 _step‚484 ' lr-AdamW/pg1‚6.0665031525807304e-05ä=Ÿ/2ð- _wandbruntime‚4412  _runtime‚4412f1Èß'ñÙO O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _runtime‚4412.419712588 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚175397782033408 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚175397595030528 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79193092 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚176983353495552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33653170 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 % _timestamp‚1.7884118020073256e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78029239 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198025586 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚176983329555456 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚568 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8523449363456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8378202827776 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78029239 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8378371827712 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78029241 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45539354 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚167237245625196 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚167237063464432 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚167019410205696 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8378202827776 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119996347 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79192524 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45539356 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78029241 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚167019392202752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33653736 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0¸Ù“fòÞO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?Devicï"`©ã%eStatsMonitor.on_train_batch_start/active.large_pool.freed‚78029239 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8378371827712 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78029241 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45539354 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚167237245625196 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚167237063464432 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚167019410205696 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8378202827776 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119996347 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79192524 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45539356 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78029241 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚167019392202752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33653736 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _runtime‚4412.419712588 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚175397782033408 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚175397595030528 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79193092 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚176983353495552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33653170 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 % _timestamp‚1.7884118020073256e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78029239 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198025586 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚176983329555456 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚568 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8523449363456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8378202827776 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944å™ãáw2ó- _wandbruntime‚4413  _runtime‚4413wíÝè&ôâM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚167264395811312 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚167046661567488 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚185537108822528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚167264577972284 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78041900 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120016134 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198078098 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198058032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158908946569216 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 trainer/global_step‚5949 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8379764979200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78041898 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚542 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚175426444549632 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8379764979200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33658511 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198078098 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚167046679570432 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _step‚486 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8355466281456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8379595965952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120036198 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8525078987264 _runtime‚4413.075523007 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚175426257533440 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79205242 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚185537345431040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚177012242503680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78041898 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8524866318848 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚158908929529856 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚167046661567488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33659053 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8355466281456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8379595965952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120036198 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8525078987264 _runtime‚4413.075523007 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚175426257533440 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79205242 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚185537345431040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚177012242503680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78041898 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8524866318848 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚158908929529856 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚167046661567488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33659053 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚167264395811312 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚167046661567488 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚185537108822528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚167264577972284 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78041900 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120016134 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198078098 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198058032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158908946569216 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 trainer/global_step‚5949 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8379764979200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78041898 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚542 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚175426444549632 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8379764979200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33658511 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0æzJ¨2ö- _wandbruntime‚4413  _runtime‚4413Ê6Õm^÷Ø -loss/FRt4/detection_loss‚2.16603762871128 % _timestamp‚1.7884118026646316e+09 1loss/FBt1/assignment_loss‚0.39584068602822037 1loss/FBt2/assignment_loss‚0.15221990414223305 1loss/FBt4/assignment_loss‚0.43967484461609274 /loss/FRt2/detection_loss‚2.4344782822188877 .loss/FBt1/detection_loss‚2.880704492330551 epoch‚4 /loss/FRt1/detection_loss‚2.9831374184838655 /loss/FRt3/detection_loss‚1.9782936972379686 1loss/SRqqt4/detection_loss‚1.7608770452580362 &loss/total_loss‚48.703385609424735 2loss/SRqqt4/assignment_loss‚1.3220314621081892 1loss/FBt3/assignment_loss‚0.19603509007288836 1loss/SRqqt2/detection_loss‚1.5191351529775243 /loss/FRt3/assignment_loss‚2.497485257387161 /loss/FBt2/detection_loss‚2.7573202550411224 /loss/FBt3/detection_loss‚1.0256200554611028 trainer/global_step‚5949 /loss/FBt4/detection_loss‚3.0507160127162933 _runtime‚4413.076659027 /loss/FRt1/assignment_loss‚4.534862255227977 /loss/FRt2/assignment_loss‚2.896267194123495 0loss/FRt4/assignment_loss‚3.1308858879541948 1loss/SRqqt1/detection_loss‚2.8772804452025373 1loss/SRqqt3/detection_loss‚1.7245412529022857 _step‚487 2loss/SRqqt1/assignment_loss‚2.5090306401252747 2loss/SRqqt2/assignment_loss‚1.6467389021012369 2loss/SRqqt3/assignment_loss‚1.8241717469963161ÓKcøÝ 1loss/SRqqt3/detection_loss‚1.7245412529022857 _step‚487 2loss/SRqqt1/assignment_loss‚2.5090306401252747 2loss/SRqqt2/assignment_loss‚1.6467389021012369 2loss/SRqqt3/assignment_loss‚1.8241717469963161 -loss/FRt4/detection_loss‚2.16603762871128 % _timestamp‚1.7884118026646316e+09 1loss/FBt1/assignment_loss‚0.39584068602822037 1loss/FBt2/assignment_loss‚0.15221990414223305 1loss/FBt4/assignment_loss‚0.43967484461609274 /loss/FRt2/detection_loss‚2.4344782822188877 .loss/FBt1/detection_loss‚2.880704492330551 epoch‚4 /loss/FRt1/detection_loss‚2.9831374184838655 /loss/FRt3/detection_loss‚1.9782936972379686 1loss/SRqqt4/detection_loss‚1.7608770452580362 &loss/total_loss‚48.703385609424735 2loss/SRqqt4/assignment_loss‚1.3220314621081892 1loss/FBt3/assignment_loss‚0.19603509007288836 1loss/SRqqt2/detection_loss‚1.5191351529775243 /loss/FRt3/assignment_loss‚2.497485257387161 /loss/FBt2/detection_loss‚2.7573202550411224 /loss/FBt3/detection_loss‚1.0256200554611028 trainer/global_step‚5949 /loss/FBt4/detection_loss‚3.0507160127162933 _runtime‚4413.076659027 /loss/FRt1/assignment_loss‚4.534862255227977 /loss/FRt2/assignment_loss‚2.896267194123495 0loss/FRt4/assignment_loss‚3.1308858879541948 1loss/SRqqt1/detection_loss‚2.8772804452025373ç8Hëù‚(:† ¦ÿãÔ©¨™ gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.993' gpu.0.enforcedPowerLimitWat“Kh¦ts‚500.0 gpu.0.powerPercent‚23.5986 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•$ÇÎ$ú‚(:™ ¦ÿãÔµŸ‡œ disk.nvme0c0n1.in‚ 710.40625' disk.nvme0c0n1.out‚61038.2216796875 cpu‚25.338367591539242 network.recv‚ 1.666063e+06) disk./.usagePercent‚73.67939639895297 disk.sda1.out‚0 disk.rbd1.out‚2.515625 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0! disk.rbd2.out‚102.14501953125 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚98.49348068237305 disk.sda3.out‚ 180.17578125& proc.memory.availableMB‚ 246.828125* proc.memory.percent‚17.568230628967285 proc.cpu.threads‚17% disk./.usageGB‚1097.8762512207031 disk.nvme0n1.in‚ 710.40625% disk.nvme0n1.out‚61038.2216796875" disk.dm-0.out‚61038.2216796875 disk.rbd0.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2878.37890625 disk.sda.out‚ 180.17578125 disk.sda1.in‚0 disk.drbd1777.out‚0 network.sent‚ 1.3527925e+07 disk.sda.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 710.40625 disk.dm-1.out‚ 180.17578125 disk.rbd3.out‚0 disk.rbd1.in‚0¥Þ›û‚(: µÿãÔÁ¢´™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.46' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.491999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çzy$ü‚(:™ µÿãÔ°Ÿßœ disk.rbd2.in‚0 disk.sda.out‚ 180.6328125 disk.sda1.out‚0 cpu‚25.337390678609683 network.sent‚ 1.363641e+07 disk.drbd1777.in‚0 disk.dm-0.in‚ 710.40625 disk.sda1.in‚0% disk./.usageGB‚1097.8762512207031# disk.dm-0.out‚61038.77099609375 disk.dm-1.in‚0) disk./.usagePercent‚73.67939639895297 disk.rbd3.in‚0 disk.sda.in‚0 disk.rbd1.out‚2.515625 disk.nvme0n1.in‚ 710.40625 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda3.out‚ 180.6328125 network.recv‚ 1.672546e+06 disk.drbd1777.out‚0 disk.dm-1.out‚ 180.6328125 disk.rbd0.in‚0! disk.rbd2.out‚102.16064453125$ memory_percent‚98.49581718444824( disk.nvme0c0n1.out‚61038.77099609375' proc.memory.availableMB‚ 246.4453125* proc.memory.percent‚17.568230628967285& disk.nvme0n1.out‚61038.77099609375 disk.nvme0c0n1.in‚ 710.40625 disk.sda3.in‚0# proc.memory.rssMB‚ 2878.37890625 disk.rbd1.in‚0Ÿ7€è2ý- _wandbruntime‚4446  _runtime‚4446X/ÕÆþÀ $ _timestamp‚1.788411836546522e+09 _runtime‚4446.958610026 _step‚488 & lr-AdamW/pg1‚5.994958772734849e-05 & lr-AdamW/pg2‚5.994958772734849e-05 trainer/global_step‚5999kºßÕËÿÅ _step‚488 & lr-AdamW/pg1‚5.994958772734849e-05 & lr-AdamW/pg2‚5.994958772734849e-05 trainer/global_step‚5999 $ _timestamp‚1.788411836546522e+09 _runtime‚4446.958610026èˆïí2€- _wandbruntime‚4446  _runtime‚4446ö‚§ÉÞ'ØO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199671278 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 $ _timestamp‚1.788411836547223e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45908204 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚160180162503680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _step‚489 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121009087 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176830734589952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚168382878442496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120989049 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚178428976966656 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8594302842368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚168603876560404 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚168603694399640 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8423548935320 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8448043150336 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚160180145464320 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121009087 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚187023522722304 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 trainer/global_step‚5999 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 199651238 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8448043150336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33926529 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚168382878442496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8447874150400 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45908206 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120989049 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8447874150400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2E´Òã'‚ÝO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚176830734589952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚168382860439552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚168382878442496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8447874150400 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45908206 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120989049 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8447874150400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199671278 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 $ _timestamp‚1.788411836547223e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45908204 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚160180162503680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _step‚489 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121009087 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176830734589952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚168382878442496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120989049 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚178428976966656 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8594302842368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚168603876560404 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚168603694399640 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8423548935320 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8448043150336 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚160180145464320 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121009087 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚187023522722304 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 trainer/global_step‚5999 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 199651238 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8448043150336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33926529 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚490 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78674850 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚168631208907492 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79846894 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8595935628288 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168631026746520 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 % _timestamp‚1.7884118371958318e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33931313 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 4.DeviceStatsMonitor.on_train_batch_end/num_oomsY°Ú®‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121008836 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199703750 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45915581 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚187053608680448 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408ÀÑÎÔî&…èM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8424938099864 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121028900 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8449267288576 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79847449 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚160206088646656 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33931313 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121008836 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199703750 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45915581 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚187053608680448 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 trainer/global_step‚5999 _runtime‚4447.608524459 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33931866 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8595718765568 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8449267288576 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚490 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78674850 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚168631208907492 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79846894 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8595935628288 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168631026746520 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 % _timestamp‚1.7884118371958318e+09 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2460 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 8529260149248 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 8529132840960 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚199 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 410712064 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 127308288 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚34415181 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚179090923473408 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚170755399152912 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚217 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 269877760 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚180719571237888 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 417333248 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚189396386296320 % _timestamp‚1.7884119159445243e+09 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 122283372 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚34415471 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚985 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚162250707981312 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚416 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚170561663324160 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 115961344 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚170561645321216 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 4912020992 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 410712064 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 123925592o•ÅT _step‚492 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 122267395 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 145311232 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 8529260149248 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 8529132840960 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚199 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ ”ÍVX7'410712064 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 127308288 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚34415181 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚179090923473408 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚170755399152912 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚217 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 269877760 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚180719571237888 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 417333248 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚189396386296320 % _timestamp‚1.7884119159445243e+09 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 122283372 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚34415471 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚985 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚162250707981312 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚416 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚170561663324160 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 115961344 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚170561645321216 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 4912020992 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 410712064 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 123925592 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2460 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚217 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 8529609285120 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚290 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚162268609534976 trainer/global_step‚6044 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 145311232 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚199 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 202054894 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚79780103 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚170580528509952 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚46572548 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚189416338997760 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚416 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚199 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 202054894 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚179110137795072 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2460 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 8505021738576 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚34420060 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚79780105 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 122290768 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 8529481976832 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 127308288 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚79780103 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚180739174803456 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚179109992483840 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚79780105 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚34420350 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 8677188134400 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 4757245257 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 139901440 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 115961344 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 5041553408 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚170580510507008 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1006 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚180739150863360 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚217 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚80992606 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚46572546 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚189416478899200 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 5041553408 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 127308288 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 410712064 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 140964952 % _timestamp‚1.7884119168708944e+09 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚179110137795072 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 417333248 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 417333248 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚170580510507008 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚26 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 _step‚493 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 4912020992 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 8529609285120 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 202070873 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚985 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 145311232 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 8529481976832 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 269877760 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 123925592 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚416 _runtime‚ 4527.28335021 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 4912020992 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 5041553408@йFT)˜ÎR YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚170580528509952 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚46572548 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚189416338997760 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚416 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚199 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 202054894 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚179110137795072 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2460 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 8505021738576 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚34420060 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚79780105 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 122290768 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 8529481976832 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 127308288 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚79780103 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚180739174803456 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚179109992483840 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚79780105 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚34420350 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 8677188134400 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 4757245257 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 139901440 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 115961344 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 5041553408 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚170580510507008 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1006 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚180739150863360 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚217 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚80992606 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚46572546 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚189416478899200 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 5041553408 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 127308288 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 410712064 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 140964952 % _timestamp‚1.7884119168708944e+09 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚179110137795072 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 417333248 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 417333248 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚170580510507008 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚26 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 _step‚493 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 4912020992 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 8529609285120 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 202070873 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚985 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 145311232 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 8529481976832 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 269877760 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 123925592 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚416 _runtime‚ 4527.28335021 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 4912020992 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 5041553408 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚170580528509952 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 202070873 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 122274791 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 8677304095744 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚170773648312912 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚80992898 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚292 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚162268626574336 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚416 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚217 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 8529609285120 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚290 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚162268609534976 trainer/global_step‚6044 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 145311232 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚199 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 202054894 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚79780103íëå1Ò™‚(:„ €äÔû§·™ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.219' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6438 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™§ËË%š‚(:š €äÔØíœ disk.sda.in‚0( disk.nvme0c0n1.out‚61437.21533203125( proc.memory.availableMB‚ 246.85546875 cpu‚25.389297930164645 disk.sda1.out‚0 disk.sda2.out‚0 network.sent‚ 1.3772962e+07 disk.dm-1.out‚ 183.9296875 disk.rbd0.out‚0 network.recv‚ 1.698926e+06 disk.dm-0.in‚ 710.40625 disk.sda2.in‚0$ memory_percent‚98.49331378936768# proc.memory.rssMB‚ 2878.37890625* proc.memory.percent‚17.568230628967285 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 710.40625 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚2.515625 disk.nvme0n1.in‚ 710.40625& disk.nvme0n1.out‚61437.21533203125 disk.rbd0.in‚0 disk.rbd2.in‚0) disk./.usagePercent‚73.67703831355507% disk./.usageGB‚1097.8411140441895 disk.rbd1.in‚0# disk.dm-0.out‚61437.21533203125 disk.dm-1.in‚0 disk.rbd2.out‚104.2275390625 disk.rbd3.in‚0 disk.sda.out‚ 183.9296875 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 183.9296875É`´£›- _wandbœÈ™ù#runtime‚4539  _runtime‚4539Ô˽Ϝإ ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 particle/accuracy_4_of_6‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_6‚0.49358975887298584 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 $ _timestamp‚1.788411929064439e+09 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.19925925135612488 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1392890363931656 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9917800426483154 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09509005397558212 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4034534692764282 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008422669023275375 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.39367812871932983 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.49043041467666626 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.46000000834465027 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_6_of_7‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.31263574957847595 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21921700239181519 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.37432950735092163 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11950428038835526 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4695417881011963 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1626984179019928 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2440333217382431 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.3888888955116272 7validation_average_jet_accuracy‚0.41265594959259033 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.420138955116272 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4037274122238159 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9914049506187439 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.30875688791275024 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.47467392683029175 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2329084724187851 *jet/accuracy_1_of_2‚0.7213672399520874 particle/accuracy_3_of_3‚1 particle/accuracy_3_of_4‚1 trainer/global_step‚6044 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 <"Purity/2FRt*SRqqt2FBt/event_purity‚0.0058139534667134285 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9864464998245239 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9425286650657654 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04768611863255501 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05813322216272354 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13392184674739838 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 particle/accuracy_1_of_5‚1 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5350877642631531 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 )loss/val_total_loss‚49.07090146183705 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4621659815311432 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8815789222717285 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02432795614004135 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4417079985141754 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3928571343421936 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2323034703731537 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29010576009750366 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.428516685962677 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3142857253551483 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 jet/accuracy_6_of_6‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.761904776096344 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02513723075389862 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2574712634086609 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.0223210658878088 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0822368711233139 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 _runtime‚4539.478621019 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.23481470346450806 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4223686158657074 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4574170708656311 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2777777910232544 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2089882493019104 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.29521289467811584 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4618328809738159 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9876921772956848 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11220572143793106 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5012295246124268 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.46661943197250366 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.15977533161640167 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9704850316047668 +jet/accuracy_2_of_4‚0.33335426449775696 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02233732119202614 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14098505675792694 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.995573878288269 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1830132007598877 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_7‚1 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.41095438599586487 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.19977478682994843 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.17959925532341003 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009542706422507763 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9785009622573853 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3808278739452362 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.984795331954956 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 *particle/sensitivity‚0.302863746881485 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1973458081483841 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1384066343307495 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5279470682144165 particle/accuracy_4_of_4‚1 particle/accuracy_3_of_6‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.48253464698791504 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5266262292861938 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2037878930568695 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.11829891055822372 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 6 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.344696968793869 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4791666567325592 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4451428949832916 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_2‚0.10740997642278671 particle/accuracy_2_of_5‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5008240342140198 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3818083107471466 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.31190475821495056 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4732142984867096 - Purity/0FRt0SRqqt3FBt/FBt_purity‚0.921875 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5078256726264954 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4416666626930237 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5166666507720947 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11393927782773972 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6778410077095032 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.4000000059604645 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99538654088974 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.30604055523872375 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9910714626312256 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 (particle/accuracy‚0.8513608574867249 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3554285168647766 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.34314924478530884 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.024420998990535736 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.23047135770320892 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9791860580444336 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 jet/accuracy_5_of_5‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3772934675216675 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4131448268890381 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18173374235630035 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.20867541432380676 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0022564581595361233 *jet/accuracy_1_of_4‚0.8854694962501526 particle/accuracy_2_of_4‚1 particle/accuracy_1_of_7‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.37030649185180664 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5277079939842224 ,jet/accuracy_4_of_5‚0.006496415939182043 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5716338157653809 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9977986216545105 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14005407691001892 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1533469706773758 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 particle/accuracy_3_of_7‚1 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0032051282469183207 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 particle/accuracy_5_of_6‚1 jet/accuracy_5_of_7‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8333333134651184 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.45084017515182495 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9909334182739258 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9928934574127197 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.023582767695188522 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 'particle/f_score‚0.3886111378669739 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2083333283662796 jet/accuracy_7_of_7‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9628472924232483 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.15909090638160706 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6436958312988281 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14131377637386322 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_7_of_7‚0.5 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.45356982946395874 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.44484469294548035 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06914380192756653 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2083333283662796 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4878842234611511 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.262037068605423 jet/accuracy_2_of_7‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9892489314079285 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4068809151649475 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3691362142562866 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_2‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.07999999821186066 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19159354269504547 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3450203835964203 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4907156229019165 + Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1875 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5446327924728394 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014264389872550964 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.00657894741743803 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_1_of_3‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13829396665096283 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4762345552444458 particle/accuracy_2_of_3‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.14814814925193787 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 epoch‚4 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.40521880984306335 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4797343909740448 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8247026801109314 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4029252529144287 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.20862069725990295 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 particle/accuracy_6_of_6‚1 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9788498282432556 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.38729479908943176 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9724438190460205 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3832429349422455 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 _step‚494 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5651050209999084 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06292631477117538 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.21111111342906952 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4531666338443756 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 *jet/accuracy_3_of_5‚0.1123143881559372 jet/accuracy_3_of_7‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16221605241298676 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9984410405158997 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_4_of_6‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4197559058666229 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13392184674739838 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5615078806877136 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4373897612094879 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.46000000834465027 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2799481153488159 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_6‚0.17307692766189575 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4586027264595032 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14668305218219757 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0028860028833150864 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004141361452639103 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5472927689552307 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 +jet/accuracy_3_of_4‚0.05368146300315857 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17492128908634186 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.21935254335403442 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_6‚0.9615384340286255 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.43628618121147156 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9990444779396057 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1280408799648285 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9834789633750916 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.17906543612480164 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14220865070819855 particle/accuracy_1_of_4‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9599999785423279 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.017924528568983078 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09491284191608429 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.34593021869659424 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.988784670829773 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 particle/accuracy_2_of_7‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9066193699836731 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.17247168719768524 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.021322930231690407 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18347442150115967 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.48716044425964355 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.29629626870155334 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4589943587779999 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4451428949832916 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9528393745422363 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9559972286224365 particle/accuracy_3_of_5‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.31264883279800415 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9856354594230652 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.049940187484025955 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3296296000480652 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.305913507938385 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/0FRt0SRqqt1FB'óêt/event_purity‚0.9984410405158997 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 )jet/accuracy_1_of_5‚0.918740451335907 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.36052629351615906 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4265178143978119 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14159861207008362 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6969696879386902 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18069516122341156 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11419752240180969 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_4_of_7‚1 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.42965570092201233 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.46211516857147217 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4864583909511566 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5685283541679382 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4583333432674408 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_6‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16187842190265656 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_6‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13823182880878448 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.13697317242622375 *jet/accuracy_1_of_3‚0.8406955599784851 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4791666567325592 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4415687620639801 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.49032846093177795 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04523809626698494 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_5‚1 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9864420294761658 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9899371266365051 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3256944417953491 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.35745614767074585 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4270833134651184 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9878824353218079 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.057060759514570236 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011294350028038025 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13475178182125092 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9839355945587158 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022fù5ŒâlÝ¥ >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.40521880984306335 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4797343909740448 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8247026801109314 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4029252529144287 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.20862069725990295 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 particle/accuracy_6_of_6‚1 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9788498282432556 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.38729479908943176 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9724438190460205 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3832429349422455 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 _step‚494 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5651050209999084 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06292631477117538 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.21111111342906952 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4531666338443756 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 *jet/accuracy_3_of_5‚0.1123143881559372 jet/accuracy_3_of_7‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16221605241298676 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9984410405158997 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_4_of_6‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4197559058666229 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13392184674739838 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5615078806877136 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4373897612094879 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.46000000834465027 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2799481153488159 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_6‚0.17307692766189575 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4586027264595032 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14668305218219757 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0028860028833150864 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004141361452639103 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5472927689552307 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 +jet/accuracy_3_of_4‚0.05368146300315857 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17492128908634186 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.21935254335403442 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_6‚0.9615384340286255 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.43628618121147156 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9990444779396057 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1280408799648285 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9834789633750916 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.17906543612480164 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14220865070819855 particle/accuracy_1_of_4‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9599999785423279 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.017924528568983078 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09491284191608429 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.34593021869659424 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.988784670829773 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 particle/accuracy_2_of_7‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9066193699836731 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.17247168719768524 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.021322930231690407 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18347442150115967 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.48716044425964355 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.29629626870155334 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4589943587779999 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4451428949832916 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9528393745422363 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9559972286224365 particle/accuracy_3_of_5‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.31264883279800415 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9856354594230652 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.049940187484025955 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3296296000480652 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.305913507938385 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9984410405158997 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 )jet/accuracy_1_of_5‚0.918740451335907 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.36052629351615906 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4265178143978119 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14159861207008362 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6969696879386902 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18069516122341156 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11419752240180969 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_4_of_7‚1 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.42965570092201233 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.46211516857147217 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4864583909511566 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5685283541679382 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4583333432674408 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_6‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16187842190265656 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_6‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13823182880878448 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.13697317242622375 *jet/accuracy_1_of_3‚0.8406955599784851 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4791666567325592 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4415687620639801 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.49032846093177795 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04523809626698494 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_5‚1 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9864420294761658 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9899371266365051 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3256944417953491 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.35745614767074585 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4270833134651184 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9878824353218079 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.057060759514570236 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011294350028038025 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13475178182125092 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9839355945587158 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 particle/accuracy_4_of_6‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_6‚0.49358975887298584 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 $ _timestamp‚1.788411929064439e+09 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.19925925135612488 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1392890363931656 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9917800426483154 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09509005397558212 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4034534692764282 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008422669023275375 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.39367812871932983 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.49043041467666626 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.46000000834465027 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_6_of_7‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.31263574957847595 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21921700239181519 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.37432950735092163 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11950428038835526 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4695417881011963 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1626984179019928 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2440333217382431 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.3888888955116272 7validation_average_jet_accuracy‚0.41265594959259033 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.420138955116272 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4037274122238159 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9914049506187439 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.30875688791275024 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.47467392683029175 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2329084724187851 *jet/accuracy_1_of_2‚0.7213672399520874 particle/accuracy_3_of_3‚1 particle/accuracy_3_of_4‚1 trainer/global_step‚6044 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 <"Purity/2FRt*SRqqt2FBt/event_purity‚0.0058139534667134285 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9864464998245239 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9425286650657654 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04768611863255501 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05813322216272354 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13392184674739838 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 particle/accuracy_1_of_5‚1 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5350877642631531 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 )loss/val_total_loss‚49.07090146183705 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4621659815311432 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8815789222717285 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02432795614004135 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4417079985141754 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3928571343421936 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2323034703731537 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29010576009750366 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.428516685962677 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3142857253551483 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 jet/accuracy_6_of_6‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.761904776096344 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02513723075389862 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2574712634086609 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.0223210658878088 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0822368711233139 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 _runtime‚4539.478621019 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.23481470346450806 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4223686158657074 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4574170708656311 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2777777910232544 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2089882493019104 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.29521289467811584 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4618328809738159 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9876921772956848 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11220572143793106 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5012295246124268 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.46661943197250366 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.15977533161640167 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9704850316047668 +jet/accuracy_2_of_4‚0.33335426449775696 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02233732119202614 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14098505675792694 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.995573878288269 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1830132007598877 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_7‚1 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.41095438599586487 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.19977478682994843 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.17959925532341003 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009542706422507763 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9785009622573853 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3808278739452362 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.984795331954956 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 *particle/sensitivity‚0.302863746881485 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1973458081483841 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1384066343307495 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5279470682144165 particle/accuracy_4_of_4‚1 particle/accuracy_3_of_6‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.48253464698791504 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5266262292861938 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2037878930568695 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.11829891055822372 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 6 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.344696968793869 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4791666567325592 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4451428949832916 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_2‚0.10740997642278671 particle/accuracy_2_of_5‚1 & Purity/1FRt3SRqqt1FBt/FBt_pDqþ&urity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5008240342140198 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3818083107471466 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.31190475821495056 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4732142984867096 - Purity/0FRt0SRqqt3FBt/FBt_purity‚0.921875 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5078256726264954 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4416666626930237 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5166666507720947 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11393927782773972 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6778410077095032 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.4000000059604645 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99538654088974 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.30604055523872375 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9910714626312256 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 (particle/accuracy‚0.8513608574867249 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3554285168647766 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.34314924478530884 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.024420998990535736 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.23047135770320892 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9791860580444336 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 jet/accuracy_5_of_5‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3772934675216675 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4131448268890381 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18173374235630035 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.20867541432380676 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0022564581595361233 *jet/accuracy_1_of_4‚0.8854694962501526 particle/accuracy_2_of_4‚1 particle/accuracy_1_of_7‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.37030649185180664 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5277079939842224 ,jet/accuracy_4_of_5‚0.006496415939182043 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5716338157653809 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9977986216545105 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14005407691001892 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1533469706773758 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 particle/accuracy_3_of_7‚1 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0032051282469183207 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 particle/accuracy_5_of_6‚1 jet/accuracy_5_of_7‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8333333134651184 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.45084017515182495 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9909334182739258 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9928934574127197 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.023582767695188522 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 'particle/f_score‚0.3886111378669739 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2083333283662796 jet/accuracy_7_of_7‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9628472924232483 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.15909090638160706 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6436958312988281 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14131377637386322 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_7_of_7‚0.5 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.45356982946395874 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.44484469294548035 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06914380192756653 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2083333283662796 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4878842234611511 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.262037068605423 jet/accuracy_2_of_7‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9892489314079285 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4068809151649475 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3691362142562866 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_2‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.07999999821186066 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19159354269504547 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3450203835964203 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4907156229019165 + Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1875 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5446327924728394 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014264389872550964 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.00657894741743803 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_1_of_3‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13829396665096283 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4762345552444458 particle/accuracy_2_of_3‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.14814814925193787 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 epoch‚4îÂ*/Æ›ž‚(: ž€äÔ‘Œ‡™ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.751' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.750200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰´¸š'Ÿ‚(:œ ž€äÔþ¯Ù›) disk./.usagePercent‚73.67708951506192 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚2.515625 disk.nvme0c0n1.in‚ 710.40625 disk.sda3.in‚0$ memory_percent‚97.71444797515869# proc.memory.rssMB‚ 2878.37890625 cpu‚25.39831011540764 network.sent‚ 1.3881175e+07 disk.sda.out‚ 184.34765625 disk.nvme0n1.in‚ 710.40625& disk.nvme0n1.out‚61659.28564453125# disk.dm-0.out‚61659.28564453125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚61659.28564453125 disk.sda2.out‚0 disk.sda3.out‚ 184.34765625( proc.memory.availableMB‚ 374.46484375 disk.rbd1.in‚0 disk.dm-0.in‚ 710.40625 disk.rbd3.in‚0* proc.memory.percent‚17.568230628967285% disk./.usageGB‚1097.8418769836426 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 184.34765625 disk.rbd2.out‚105.9423828125 disk.sda.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.drbd1777.out‚0 network.recv‚ 1.705432e+06´C‡g2 - _wandbruntime‚4545  _runtime‚4545šOÒÅ¡¿ _step‚495 & lr-AdamW/pg1‚5.923217827251144e-05 & lr-AdamW/pg2‚5.923217827251144e-05 trainer/global_step‚6049 # _timestamp‚1.78841193465376e+09 _runtime‚4545.065866106Øõå梀 # _timestamp‚1.78841193465376e+09 _runtime‚4545.065866106 _step‚495 & lr-AdamW/pg1‚5.923217827251144e-05 & lr-AdamW/pg2‚5.923217827251144e-05 trainer/global_step‚6049ïÀ4-2£- _wandbruntime‚4545  _runtime‚4545®` ²ß'¤ÙO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179474393404928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170934853384192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚170934835381248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚6049 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚4545.066838558 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79969997 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202419631 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202439671 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8687379226624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚181109672948736 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚171120039522226 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122449636 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79969995 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189797271148544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚181109649008640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884119346545064e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170934835381248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81190604 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚171120221682990 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79969997 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179474393404928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34504048 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚162605119501312 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚179474580407808 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170934853384192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122469674 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34503578 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189797028235264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79969995 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46686556 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚179474580407808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81190132 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122449636u-Iþä'¥ÞO E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122449636 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179474393404928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170934853384192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚170934835381248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚6049 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚4545.066838558 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79969997 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202419631 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202439671 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8687379226624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚181109672948736 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚171120039522226 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122449636 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79969995 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189797271148544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚181109649008640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884119346545064e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170934835381248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81190604 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚171120221682990 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79969997 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179474393404928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34504048 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚162605119501312 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚179474580407808 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170934853384192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122469674 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34503578 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189797028235264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79969995 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46686556 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚179474580407808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81190132 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404ð\­µò2¦- _wandbruntime‚4545  _runtime‚4545ú›+9§ãM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46693933 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚162631062683648 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 REDeviceStB_â$atsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79982656 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 trainer/global_step‚6049 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 247094272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202472143 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202452077 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚189827350174208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8516326224818 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8540951161856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚162631045644288 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79982656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170962104745984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81202798 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚171147554030078 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79982654 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170962122748928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚179503055907840 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8540951161856 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81203293 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170962104745984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚495 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 % _timestamp‚1.7884119353148766e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202472143 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8541120175104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8688788217344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202452077 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚497 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122469423 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122469423 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚493 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8689011371520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064q”î&¨èM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46693933 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚162631062683648 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79982656 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 trainer/global_step‚6049 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 247094272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202472143 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202452077 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚189827350174208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8516326224818 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8540951161856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚162631045644288 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79982656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170962104745984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81202798 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚171147554030078 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79982654 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170962122748928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚179503055907840 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8540951161856 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81203293 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170962104745984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚495 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 % _timestamp‚1.7884119353148766e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202472143 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8541120175104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8688788217344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202452077 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚497 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122469423 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122469423 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚493 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8689011371520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217ñ¹8I2©- _wandbruntime‚4545  _runtime‚4545ØÔiYªÓ /loss/FRt1/assignment_loss‚5.471599159569576 0loss/FBt1/assignment_loss‚0.3583995738699741 .loss/FBt1/detection_loss‚2.505764877423644 trainer/global_step‚6049 _step‚498 1loss/SRqqt4/detection_loss‚1.8651691041094192 /loss/FBt2/detection_loss‚4.0904721975326535 /loss/FRt2/assignment_loss‚3.218289469679197 .loss/FRt4/detection_loss‚2.226435027519862 1loss/SRqqt3/detection_loss‚1.4477459415793419 epoch‚5 0loss/FRt3/assignment_loss‚2.1614520102739334 2loss/SRqqt4/assignment_loss‚1.8001705991460921 /loss/FRt1/detection_loss‚2.8666578695691864 .loss/FRt3/detection_loss‚2.022554388269782 _runtime‚ 4545.72904754 2loss/SRqqt1/assignment_loss‚2.3699968182123623 2loss/SRqqt3/assignment_loss‚1.7143951913407236 % _timestamp‚1.7884119353169818e+09 1loss/FBt3/assignment_loss‚0.21014016385858889 /loss/FBt3/detection_loss‚0.6451685408231779 /loss/FBt4/detection_loss‚2.3886659122430363 &loss/total_loss‚51.682278338374694 /loss/FRt4/assignment_loss‚3.869666784339481 0loss/FBt4/assignment_loss‚0.3869646076156222 /loss/FRt2/detection_loss‚2.4087633142868676 0loss/SRqqt1/detection_loss‚2.830291142830482 1loss/SRqqt2/detection_loss‚1.7541607885218378 2loss/SRqqt2/assignment_loss‚1.5395072635207603 0loss/FBt2/assignment_loss‚1.5298475922390935ŸpD÷^«Ø epoch‚5 0loss/FRt3/assignment_loss‚2.1614520102739334 2loss/SRqqt4/assignment_loss‚1.8001705991460921 /loss/FRt1/detection_loss‚2.8666578695691864 .loss/FRt3/detection_loss‚2.022554388269782 _runtime‚ 4545.72904754 2loss/SRqqt1/assignment_loss‚2.3699968182123623 2loss/SRqqt3/assignment_loss‚1.7143951913407236 % _timestamp‚1.7884119353169818e+09 1loss/FBt3/assignment_loss‚0.21014016385858889 /loss/FBt3/detection_loss‚0.6451685408231779 /loss/FBt4/detection_loss‚2.3886659122430363 &loss/total_loss‚51.682278338374694 /loss/FRt4/assignment_loss‚3.869666784339481 0loss/FBt4/assignment_loss‚0.3869646076156222 /loss/FRt2/detection_loss‚2.4087633142868676 0loss/SRqqt1/detection_loss‚2.830291142830482 1loss/SRqqt2/detection_loss‚1.7541607885218378 2loss/SRqqt2/assignment_loss‚1.5395072635207603 0loss/FBt2/assignment_loss‚1.5298475922390935 /loss/FRt1/assignment_loss‚5.471599159569576 0loss/FBt1/assignment_loss‚0.3583995738699741 .loss/FBt1/detection_loss‚2.505764877423644 trainer/global_step‚6049 _step‚498 1loss/SRqqt4/detection_loss‚1.8651691041094192 /loss/FBt2/detection_loss‚4.0904721975326535 /loss/FRt2/assignment_loss‚3.218289469679197 .loss/FRt4/detection_loss‚2.226435027519862 1loss/SRqqt3/detection_loss‚1.4477459415793419òC‘둬‚(:† ­€äÔ©¬š™ gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.398' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç¢‹­‚(:‘ ­€äÔ¥€·œ# disk./.usageGB‚1097.84175491333 disk.sda2.in‚0 disk.sda.in‚0 disk.rbd1.out‚2.515625# proc.memory.rssMB‚ 2878.37890625 disk.dm-1.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 184.8984375' proc.memory.availableMB‚ 374.4921875 proc.cpu.threads‚17 disk.dm-0.in‚ 710.40625 disk.rbd2.in‚0 disk.sda.out‚ 184.8984375) disk./.usagePercent‚73.67708132282084$ disk.nvme0n1.out‚61758.513671875 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 network.sent‚ 1.4005042e+07 disk.rbd2.out‚105.9423828125 disk.drbd1777.in‚0$ memory_percent‚97.71428108215332 cpu‚25.397162261246653 network.recv‚ 1.711682e+06 disk.nvme0n1.in‚ 710.40625! disk.dm-0.out‚61758.513671875 disk.nvme0c0n1.in‚ 710.40625& disk.nvme0c0n1.out‚61758.513671875 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 184.8984375 disk.rbd3.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.568230628967285¡OÏ(:„ ¼€äÔõÕ—™ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.29' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.058 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E#¨¯‚(:Ž ¼€äÔŠýØ› disk.drbd1777.in‚0$ memory_percent‚97.71506786346436( proc.memory.availableMB‚ 374.36328125 proc.cpu.threads‚17& disk.nvme0c0n1.out‚61881.900390625 disk.rbd1.in‚0 disk.nvme0n1.in‚ 710.40625 disk.dm-1.out‚ 185.40625# proc.memory.rssMB‚ 2878.37890625 network.sent‚ 1.4112617e+07 network.recv‚ 1.717364e+06 disk.sda2.in‚0 disk.dm-0.in‚ 710.40625 disk.rbd2.in‚0 disk.rbd3.in‚0 cpu‚25.39603839650879) disk./.usagePercent‚73.67705060191673 disk.nvme0c0n1.in‚ 710.40625 disk.sda3.out‚ 185.40625% disk./.usageGB‚1097.8412971496582$ disk.nvme0n1.out‚61881.900390625 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.rbd2.out‚106.04443359375 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd1.out‚2.515625! disk.dm-0.out‚61881.900390625* proc.memory.percent‚17.568230628967285 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 185.40625)?‹u2°- _wandbruntime‚4579  _runtime‚4579 ½0rȱ _step‚499 ' lr-AdamW/pg1‚5.8512952668351234e-05 ' lr-AdamW/pg2‚5.8512952668351234e-05 trainer/global_step‚6099 $ _timestamp‚1.788411969378227e+09 _runtime‚4579.790337566:bcUÍ²Ç ' lr-AdamW/pg1‚5.8512952668351234e-05 ' lr-AdamW/pg2‚5.8512952668351234e-05 trainer/global_step‚6099 $ _timestamp‚1.788411969378227e+09 _runtime‚4579.790337566 _step‚499ó]H?2³- _wandbruntime‚4579  _runtime‚4579Å-:~—´ØO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚172298321620992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚182555320359936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚6099 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚526 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚172298303618048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8758431938560 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80602947 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80602945 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚180907532964352 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123462376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47055406 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47055404 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123462376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚180907532964352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81826970 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚4579.791348791 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8609229346304 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚524 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚172486670457434 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8584574000086 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 $ _timestamp‚1.788411969378945e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123442338 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81826444 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 _step‚500 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8609398346240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8609229346304 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123442338 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚191313513579520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182555296419840 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880(†ÅòG  WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80602945 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204045283 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34771040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚180907719967232 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚172298303618048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191313752298496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚163902261578752 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8609229346304 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123442338 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚191313513579520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182555296419840 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80602945 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204045283 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34771040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚180907719967232 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚172298303618048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191313752298496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚163902261578752 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚180907532964352 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123462376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47055406 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47055404 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123462376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚180907532964352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81826970 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚4579.791348791 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8609229346304 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚524 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚172486670457434 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0ôpTª2¶- _wandbruntime‚4580  _runtime‚4580áŽÚöé&·ãM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8610791497728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204077729 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81838997 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182584233308160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8610791497728 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 trainer/global_step‚6099 _step‚501 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8610622484480 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204097795 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34776216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204077729 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8610622484480 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚180936382483456 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚172325572982784 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚191344079974912 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚182584209368064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚172514002804314 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8585798043226 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123462125 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204097795 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚163928221800448 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8585963164838 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34776728è@çy˜$¸èM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚191343851755008 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚172325572982784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47062783 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228219904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚180936382483456 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123482189 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80615604 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚172514184965286 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123462125 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123482189 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81839511 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 trainer/global_step‚6099 _step‚501 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8610622484480 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204097795 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34776216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204077729 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8610622484480 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚180936382483456 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚172325572982784 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚191344079974912 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚182584209368064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚172514002804314 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8585798043226 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123462125 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204097795 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚163928221800448 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8585963164838 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34776728 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8610791497728 H:DeviceStatsMonitor.on_trŽðÊ–Vain_batch_end/allocation.all.freed‚ 204077729 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81838997 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182584233308160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8610791497728 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚182340672523776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35039381 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚173853483553406 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47424256 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81235895 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884120041697977e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205670935 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8829120120832 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚165199420695552 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚182340859526656 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚173661789857792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184000967771136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8829332802560 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚173853301392642 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0ªØæ¡â'ÅÜO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82463637 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124435040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8678900668928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8678900668928 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35038845 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205690975 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47424254 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124455078 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8679069668864 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚173853483553406 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47424256 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81235895 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884120041697977e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205670935 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8829120120832 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚165199420695552 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚182340859526656 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚173661789857792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184000967771136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8829332802560 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚173853301392642 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚182340672523776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35039381 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚165225380917248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8655434983246 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82475729 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205703381 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81248556 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚182369335026688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205723447 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 trainer/global_step‚6149 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚173689059222528 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚182369522042880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184029880719360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8680293807104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚173880633739522 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205703381 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8680462820352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47431631 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 _runtime‚4615.236485338 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81248554 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173689041219584 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8680462820352 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 % _timestamp‚1.7884120048239448e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚182369522042880 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚P¡Ïªã23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8655269861634 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205723447 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124474891 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82476266 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124474891 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124454827 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8680293807104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚173689041219584 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8830750924288 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚192860392937984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81248556 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47431633 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35044098 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0„3 î&ÈèM H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205703381 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81248556 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚182369335026688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205723447 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 trainer/global_step‚6149 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚173689059222528 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚182369522042880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184029880719360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8680293807104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚173880633739522 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205703381 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8680462820352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47431631 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 _runtime‚4615.236485338 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81248554 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173689041219584 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8680462820352 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 % _timestamp‚1.7884120048239448e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚182369522042880 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8655269861634 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205723447 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124474891 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82476266 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124474891 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124454827 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8680293807104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚173689041219584 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8830750924288 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚192860392937984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81248556 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47431633 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35044098 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚165225380917248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8655434983246 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82475729 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006ùL¤0g2É- _wandbruntime‚4615  _runtime‚46153‹ž´WÊÑ 0loss/FRt3/assignment_loss‚2.2247983239315174 2loss/SRqqt1/assignment_loss‚2.3657998894823007 2loss/SRqqt2/assignment_loss‚1.5695576759587442 /loss/SRqqt1/detection_loss‚2.68068612649523 epoch‚5 2loss/SRqqt4/assignment_loss‚1.5530918172972148 /loss/FRt4/detection_loss‚2.2554518042826186 0loss/SRqqt2/detection_loss‚1.560241265807833 .loss/FRt1/detection_loss‚3.028940191268921 .loss/FRt2/detection_loss‚2.181028122703234 1loss/FBt2/assignment_loss‚0.14221037924289703 0loss/FBt3/assignment_loss‚0.3015327685360505 /loss/FRt3/detection_loss‚1.9655550494790077 1loss/SRqqt4/detection_loss‚1.9902156224084455 /loss/FRt4/assignment_loss‚3.865679640395969 1loss/SRqqt3/assignment_loss‚1.495208742140676 /loss/FBt1/detection_loss‚2.7669194489717484 % _timestamp‚1.7884120048259292e+09 _runtime‚4615.237979713 0loss/FBt1/assignment_loss‚0.5464749425167005 0loss/FBt4/assignment_loss‚0.6050065723159017 &loss/total_loss‚51.120922664907425 _step‚506 1loss/SRqqt3/detection_loss‚1.3173018008237705 .loss/FBt2/detection_loss‚6.278821051120758 /loss/FBt3/detection_loss‚0.7649346083045416 trainer/global_step‚6149 /loss/FRt1/assignment_loss‚4.739630813598633 /loss/FRt2/assignment_loss‚3.028826590627432 /loss/FBt4/detection_loss‚1.8930094171972836I*÷!\ËÖ 1loss/FBt2/assignment_loss‚0.14221037924289703 0loss/FBt3/assignment_loss‚0.3015327685360505 /loss/FRt3/detection_loss‚1.9655550494790077 1loss/SRqqt4/detection_loss‚1.9902156224084455 /loss/FRt4/assignment_loss‚3.865679640395969 1loss/SRqqt3/assignment_loss‚1.495208742140676 /loss/FBt1/detection_loss‚2.7669194489717484 % _timestamp‚1.7884120048259292e+09 _runtime‚4615.237979713 0loss/FBt1/assignment_loss‚0.5464749425167005 0loss/FBt4/assignment_loss‚0.6050065723159017 &loss/total_loss‚51.120922664907425 _step‚506 1loss/SRqqt3/detection_loss‚1.3173018008237705 .loss/FBt2/detection_loss‚6.278821051120758 /loss/FBt3/detection_loss‚0.7649346083045416 trainer/global_step‚6149 /loss/FRt1/assignment_loss‚4.739630813598633 /loss/FRt2/assignment_loss‚3.028826590627432 /loss/FBt4/detection_loss‚1.8930094171972836 0loss/FRt3/assignment_loss‚2.2247983239315174 2loss/SRqqt1/assignment_loss‚2.3657998894823007 2loss/SRqqt2/assignment_loss‚1.5695576759587442 /loss/SRqqt1/detection_loss‚2.68068612649523 epoch‚5 2loss/SRqqt4/assignment_loss‚1.5530918172972148 /loss/FRt4/detection_loss‚2.2554518042826186 0loss/SRqqt2/detection_loss‚1.560241265807833 .loss/FRt1/detection_loss‚3.028940191268921 .loss/FRt2/detection_loss‚2.181028122703234ú2̂(: é€äÔ‰¬¿™ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.188' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.637600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o–„|(Í‚(: é€äÔÒÌñ›) disk./.usagePercent‚73.67707927476056 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda3.in‚0 network.sent‚ 1.4231891e+07 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 710.40625 disk.sda3.out‚ 187.02734375* proc.memory.percent‚17.568230628967285 cpu‚25.392629810610842 proc.cpu.threads‚17$ disk./.usageGB‚1097.841724395752 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 710.40625& disk.nvme0n1.out‚62180.32568359375 disk.rbd2.in‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2878.37890625 disk.rbd1.out‚2.515625# disk.dm-0.out‚62180.32568359375 disk.sda2.out‚0 network.recv‚ 1.730992e+06 disk.dm-1.out‚ 187.02734375 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.rbd2.out‚106.19287109375 disk.nvme0c0n1.in‚ 710.40625( disk.nvme0c0n1.out‚62180.32568359375 disk.sda.out‚ 187.02734375 disk.drbd1777.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 374.02734375 disk.rbd1.in‚0 disk.rbd3.out‚0$ memory_percent‚97.71711826324463>ûj΂(:„ ø€äÔ÷ߤ™ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.29' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.058 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qFaä(Ï‚(: ø€äÔ󒺛# disk.dm-0.out‚62180.68896484375# proc.memory.rssMB‚ 2878.37890625 cpu‚25.391585359117833) disk./.usagePercent‚73.67707927476056 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd2.in‚0( proc.memory.availableMB‚ 374.01953125$ disk./.usageGB‚1097.841724395752& disk.nvme0n1.out‚62180.68896484375 disk.dm-1.out‚ 187.70703125! disk.rbd2.out‚107.77880859375 disk.rbd3.in‚0$ memory_percent‚97.71716594696045 disk.nvme0c0n1.in‚ 710.40625 disk.sda3.out‚ 187.70703125 proc.cpu.threads‚17 network.recv‚ 1.736714e+06 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 187.70703125 disk.drbd1777.out‚0 disk.dm-0.in‚ 710.40625 disk.rbd0.out‚0 disk.rbd3.out‚0* proc.memory.percent‚17.568230628967285 network.sent‚ 1.4339913e+07 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 710.40625 disk.sda2.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚62180.68896484375 disk.rbd1.in‚0 disk.rbd1.out‚2.515625œ8äû2Ð- _wandbruntime‚4649  _runtime‚4649C —ÆÇÑÁ trainer/global_step‚6199 % _timestamp‚1.7884120386871455e+09 _runtime‚4649.099296192 _step‚507 & lr-AdamW/pg1‚5.706965290145989e-05 & lr-AdamW/pg2‚5.706965290145989e-05%e»ÌÒÆ & lr-AdamW/pg1‚5.706965290145989e-05 & lr-AdamW/pg2‚5.706965290145989e-05 trainer/global_step‚6199 % _timestamp‚1.7884120386871455e+09 _runtime‚4649.099296192 _step‚507ûèO}±2Ó- _wandbruntime‚4649  _runtime‚4649K7V£ß'ÔÙO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8723517636902 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125447780 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚183773999086080 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47793106 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚175025258094592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚175025240091648 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚166496596851712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35307375 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8748571991552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6199 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8748740991488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8748571991552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207316627 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8900007520256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47793104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35307887 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚166496579812352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81868847 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8900220201984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81868847 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚185446591242240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83100993 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚194346598762496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81868845 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81868845 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183773812083200 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125447780}5ù‘ÕÞO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8748571991552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207316627 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8900007520256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47793104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35307887 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚166496579812352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81868847 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8900220201984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81868847 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚185446591242240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚9$T>/S!85 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83100993 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚194346598762496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81868845 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81868845 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183773812083200 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125447780 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8723517636902 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125447780 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚183773999086080 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47793106 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚175025258094592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚175025240091648 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚166496596851712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35307375 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8748571991552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6199 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8748740991488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0ütÖÖn2Ö- _wandbruntime‚4649  _runtime‚4649Ô.žÎè&×âM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8749965129728 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207329033 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8901635175936 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81881506 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35312600 trainer/global_step‚6199 $ _timestamp‚1.788412039338603e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚183802661602304 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚175052509456384 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 125447529 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47800481 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚185475528130560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8750134142976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8724906801654 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207329033 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚183802661602304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8901422507520 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125447529 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8750134142976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚185475504190464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚166522522994688 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚175247446835702 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83113605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35313122 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚175247264674730 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚175052509456384 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚175052527459328 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚166522540034048 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81881504 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125467593 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125467593 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83113081 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚183802474586112 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8749965129728 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81881506 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚194376926697984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8724741680042 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 _runtime‚4649.751349454 _step‚509 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚175052527459328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81881504 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207349099 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207349099 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8724741680042 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 _runtime‚4649.751349454 _step‚509 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚175052527459328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81881504 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207349099 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207349099 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81881506 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35312600 trainer/global_step‚6199 $ _timestamp‚1.788412039338603e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚183802661602304 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚175052509456384 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 125447529 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47800481 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚185475528130560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8750134142976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8724906801654 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207329033 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚183802661602304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8901422507520 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125447529 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8750134142976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚185475504190464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚166522522994688 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚175247446835702 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83113605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35313122 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚175247264674730 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚175052509456384 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚175052527459328 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚166522540034048 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81881504 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125467593 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125467593 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83113081 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚183802474586112 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8749965129728 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81881506 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚194376926697984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0ý‘z[Õ2Ù- _wandbruntime‚4649  _runtime‚4649ðWèzUÚÏ 1loss/SRqqt4/assignment_loss‚1.902629216817709 /loss/FRt1/detection_loss‚3.0485146325081587 .loss/FRt4/detection_loss‚2.259769103964981 /loss/FBt1/detection_loss‚2.7845702950771036 /loss/FBt3/detection_loss‚0.8306373529625893 _runtime‚4649.752729864 .loss/FRt4/assignment_loss‚2.94500645934319 /loss/FRt2/detection_loss‚2.3303221984343097 /loss/FBt4/detection_loss‚3.2266314029693604 /loss/FRt2/assignment_loss‚2.778055892207406 1loss/FBt2/assignment_loss‚0.39655543310679303 1loss/FBt3/assignment_loss‚0.30129534992004714 0loss/FBt4/assignment_loss‚0.3983046155226475 1loss/SRqqt1/assignment_loss‚2.954635988582264 2loss/SRqqt2/assignment_loss‚1.4609393990786093 % _timestamp‚1.7884120393407006e+09 _step‚510 /loss/FRt1/assignment_loss‚4.853177838027477 2loss/SRqqt3/assignment_loss‚1.5300988524051553 0loss/FBt1/assignment_loss‚0.5200451897228329 1loss/SRqqt3/detection_loss‚1.8989699068902568 .loss/FBt2/detection_loss‚3.796866556008657 epoch‚5 trainer/global_step‚6199 ,loss/FRt3/detection_loss‚2.4709448564129 1loss/SRqqt1/detection_loss‚2.7712340598756615 /loss/FRt3/assignment_loss‚2.713364506379152 1loss/SRqqt2/detection_loss‚1.5191069406928954 0loss/SRqqt4/detection_loss‚1.578815012711745 &loss/total_loss‚51.270491059621904«µeòZÛÔ 1loss/SRqqt4/assignment_loss‚1.902629216817709 /loss/FRt1/detection_loss‚3.0485146325081587 .loss/FRt4/detection_loss‚2.259769103964981 /loss/FBt1/detection_loss‚2.7845702950771036 /loss/FBt3/detection_loss‚0.8306373529625893 _runtime‚4649.752729864 .loss/FRt4/assignment_loss‚2.94500645934319 /loss/FRt2/detection_loss‚2.3303221984343097 /loss/FBt4/detection_loss‚3.2266314029693604 /loss/FRt2/assignment_loss‚2.778055892207406 1loss/FBt2/assignment_loss‚0.39655543310679303 1loss/FBt3/assignment_loss‚0.30129534992004714 0loss/FBt4/assignment_loss‚0.3983046155226475 1loss/SRqqt1/assignment_loss‚2.954635988582264 2loss/SRqqt2/assignment_loss‚1.4609393990786093 % _timestamp‚1.7884120393407006e+09 _step‚510 /loss/FRt1/assignment_loss‚4.853177838027477 2loss/SRqqt3/assignment_loss‚1.5300988524051553 0loss/FBt1/assignment_loss‚0.5200451897228329 1loss/SRqqt3/detection_loss‚1.8989699068902568 .loss/FBt2/detection_loss‚3.796866556008657 epoch‚5 trainer/global_step‚6199 ,loss/FRt3/detection_loss‚2.4709448564129 1loss/SRqqt1/detection_loss‚2.7712340598756615 /loss/FRt3/assignment_loss‚2.713364506379152 1loss/SRqqt2/detection_loss‚1.5191069406928954 0loss/SRqqt4/detection_loss‚1.578815012711745 &loss/total_loss‚51.270491059621904þû«QQ‘Ü‚(:† ‡äÔÔè²™ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚123.507' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=¤,õÝ‚(:– ‡äÔÄîñ›! disk.rbd2.out‚107.85693359375 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda.in‚0 disk.sda.out‚ 188.390625 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda3.out‚ 188.390625$ memory_percent‚97.71571159362793 disk.dm-0.in‚ 710.40625 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2878.37890625* proc.memory.percent‚17.568230628967285% disk./.usageGB‚1097.8417854309082& disk.nvme0n1.out‚62279.89404296875# disk.dm-0.out‚62279.89404296875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 cpu‚25.390430814084684 disk.rbd1.out‚2.515625 disk.rbd3.in‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 710.40625 disk.dm-1.out‚ 188.390625( disk.nvme0c0n1.out‚62279.89404296875 network.sent‚ 1.4345387e+07 network.recv‚ 1.740322e+06' proc.memory.availableMB‚ 374.2578125 proc.cpu.threads‚ê‚\,17( disk./.usagePercent‚73.6770833708811¹g¼-‘Þ‚(:† –äÔ¢°“™ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.619' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CŽuß‚(:” –äÔ§ùØœ disk.rbd3.out‚0 disk.sda.in‚0* proc.memory.percent‚17.568230628967285 disk.dm-1.out‚ 188.84375( disk.nvme0c0n1.out‚62379.18310546875 disk.sda3.in‚0% disk./.usageGB‚1097.8418426513672 disk.sda2.in‚0 disk.rbd1.in‚0$ memory_percent‚97.71609306335449) disk./.usagePercent‚73.67708721099412 disk.dm-1.in‚0 disk.rbd2.in‚0! disk.rbd2.out‚107.87255859375 disk.sda3.out‚ 188.84375 disk.sda1.in‚0 disk.rbd1.out‚2.515625' proc.memory.availableMB‚ 374.1953125 network.sent‚ 1.4454065e+07 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda.out‚ 188.84375 disk.sda1.out‚0 proc.cpu.threads‚17 network.recv‚ 1.747102e+06 disk.drbd1777.in‚0 disk.drbd1777.out‚0& disk.nvme0n1.out‚62379.18310546875# disk.dm-0.out‚62379.18310546875 disk.rbd3.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2878.37890625 cpu‚25.389332977588253 disk.nvme0n1.in‚ 710.40625 disk.dm-0.in‚ 710.40625‹¤ïß‘à‚(:† ¥äÔ¹û—™ gpu.0.gpu‚44.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.184' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ê¨Xá‚(:Š ¥äÔâÍÈ›) disk./.usagePercent‚73.67718014172905 disk.nvme0n1.in‚ 710.40625 disk.dm-1.in‚0% disk.nvme0c0n1.out‚62380.35546875 disk.sda.out‚ 189.421875# disk.nvme0n1.out‚62380.35546875 disk.dm-0.out‚62380.35546875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 189.421875 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0$ memory_percent‚98.46963882446289# proc.memory.rssMB‚ 2878.37890625 disk.nvme0c0n1.in‚ 710.40625 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.sda1.out‚0 network.recv‚ 1.751565e+06 disk.dm-0.in‚ 710.40625 disk.rbd3.in‚0 network.sent‚ 1.4460132e+07 disk.rbd1.out‚2.515625& proc.memory.availableMB‚ 250.734375* proc.memory.percent‚17.568230628967285 cpu‚25.38826364721069 proc.cpu.threads‚17 disk.dm-1.out‚ 189.421875 disk.rbd3.out‚0 disk.sda.in‚0% disk./.usageGB‚1097.8432273864746 disk.rbd0.in‚0 disk.rbd2.out‚ 108.021484375 disk.sda3.in‚0 disk.drbd1777.out‚0Ü ê`2â- _wandbruntime‚4683  _runtime‚4683rv±¡Åã¿ _runtime‚4683.958715179 _step‚511 % lr-AdamW/pg1‚5.63458795202369e-05 % lr-AdamW/pg2‚5.63458795202369e-05 trainer/global_step‚6249 % _timestamp‚1.7884120735465858e+09¸-•ÊäÄ % lr-AdamW/pg1‚5.63458795202369e-05 % lr-AdamW/pg2‚5.63458795202369e-05 trainer/global_step‚6249 % _timestamp‚1.7884120735465858e+09 _runtime‚4683.958715179 _step‚511ÿ'È€z2å- _wandbruntime‚4683  _runtime‚4683ú Óß'æÙO O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195863327021568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚176586563263058 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 208922239 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4683.959694524 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126420444 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221070336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8970843357696 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82501797 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚186892262593536 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83739396 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚176586745423822 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185207138645504 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8792824333906 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚167793738929152 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126420444 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚176388708328448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8818243314176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8792989455310 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _step‚512 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35577442 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245010432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚6249 % _timestamp‚1.7884120735473502e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚185206951642624 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35577976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚167793755968512 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185206951642624 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚176388726331392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126440482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 208942279 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83739932 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚176388708328448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8818412314112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8818243314176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82501797 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82501795 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82501795 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185207138645504 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248ÓbÜ‹ä'çÞO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8792989455310 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _step‚512 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35577442 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245010432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚6249 % _timestamp‚1.7884120735473502e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚185206951642624 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35577976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚167793755968512 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185206951642624 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚176388726331392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126440482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 208942279 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83739932 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚176388708328448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8818412314112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8818243314176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82501797 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82501795 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82501795 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185207138645504 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195863327021568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚176586563263058 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 208922239 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4683.959694524 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126420444 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221070336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8970843357696 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82501797 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚186892262593536 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83739396 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚176586745423822 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185207138645504 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8792824333906 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚167793738929152 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126420444 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚176388708328448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8818243314176€•ì{2è- _wandbruntime‚4684  _runtime‚4684£¹;"éãM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚176415995696128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚176614077770910 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82514454 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126440231 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8819636452352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48169333 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚167819699150848 _runtime‚4684.611903982 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195893410767872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚176415977693184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8794378620062 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195893651570688 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126460295 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83753197 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8819636452352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48169331 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚186921151601664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185235801161728 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚186921175541760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126460295 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82514454 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8819805465600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35583355 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚509 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8972476028928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _step‚513 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚176415977693184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8794213498450 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208974751 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚511 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚167819682111488 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚176415995696128 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82514456 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185235614145536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83752686 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35583864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚176613895609938 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8819805465600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 trainer/global_step‚6249 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208954685 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚176415977693184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8794378620062 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195893651570688 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126460295 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83753197 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8819636452352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48169331 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚186921151601664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185235801161728 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚186921175541760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126460295 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82514454 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8819805465600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35583355 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚509 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8972476028928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _step‚513 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚176415977693184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8794213498450 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208974751 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚511 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚167819682111488 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚176415995696128 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82514456 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185235614145536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83752686 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35583864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚176613895609938 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8819805465600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 trainer/global_step‚6249 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208954685 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127433184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9041977222656 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84377373 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9041762443776 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 210567931 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚197379648508416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210547891 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚177953376359030 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚197379887227392 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _runtime‚4718.638539563 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127413146 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚177752176565248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83134745 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚186640091202048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 trainer/global_step‚6299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚169090915085312 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8887914636800 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83134745 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48530806 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8862296152314 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8888083636736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169090898045952 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8888083636736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8862461273718 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83134747 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚186640278204928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚186640091202048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚177752176565248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35846569 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚177752194568192Ǭ÷•Ë÷ÞO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚177953376359030 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚197379887227392 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _runtime‚4718.638539563 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127413146 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚177752176565248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83134745 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚186640091202048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 trainer/global_step‚6299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚169090915085312 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8887914636800 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83134745 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48530806 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8862296152314 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8888083636736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169090898045952 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8888083636736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ^LDeviceStatsMonE~Ýitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8862461273718 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83134747 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚186640278204928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚186640091202048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚177752176565248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35846569 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚177752194568192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚177752194568192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84377905 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48530804 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚188337910004736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚177953194198266 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127433184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9041977222656 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84377373 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9041762443776 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 210567931 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚197379648508416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210547891„®–³V2ø- _wandbruntime‚4719  _runtime‚4719;J7é&ùãM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83147404 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210600403 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127432933 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8889476788224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚197410210813952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8863850438470 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚177779463932928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚177779445929984 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7884121088715458e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127432933 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9043179386880 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83147406 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚177779463932928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8889307774976 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚186668940721152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 210580337 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _step‚517 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8863685316858 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 210580337 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84390153 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚177980526545146 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8889476788224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83147406 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48538181 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83147404 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127452997 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9043179386880 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83147406 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚177779463932928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8889307774976 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚186668940721152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 210580337 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _step‚517 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8863685316858 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 210580337 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84390153 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚177980526545146 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8889476788224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83147406 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48538181 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83147404 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127452997 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210600403 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127432933 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8889476788224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚197410210813952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8863850438470 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚177779463932928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚177779445929984 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7884121088715458e+09…Úá*2û- _wandbruntime‚4719  _runtime‚4719 ‡OëVüÐ _step‚518 2loss/SRqqt3/assignment_loss‚1.2020070355013013 0loss/FBt1/assignment_loss‚0.6420499769465096 /loss/FRt4/assignment_loss‚4.040292457171849 1loss/SRqqt2/assignment_loss‚1.613871421230336 1loss/SRqqt2/detection_loss‚1.7226955384016036 0loss/FBt4/assignment_loss‚0.4066797151616894 0loss/SRqqt1/detection_loss‚2.861761924766359 .loss/FBt1/detection_loss‚2.259941586426326 /loss/FBt3/detection_loss‚0.8060408568081374 %loss/total_loss‚48.71877260474845 trainer/global_step‚6299 /loss/FRt1/assignment_loss‚4.860773298475477 /loss/FBt2/assignment_loss‚0.574811094996312 1loss/FBt3/assignment_loss‚0.16714004234806004 1loss/SRqqt4/detection_loss‚1.8091506523654817 /loss/FBt2/detection_loss‚2.9091262102127073 epoch‚5 1loss/SRqqt1/assignment_loss‚2.610520053477514 2loss/SRqqt4/assignment_loss‚1.8342305287084681 1loss/SRqqt3/detection_loss‚1.5888608396053314 _runtime‚4719.285546093 .loss/FRt1/detection_loss‚3.245668846148032 /loss/FRt3/detection_loss‚1.8250546356042225 % _timestamp‚1.7884121088735106e+09 /loss/FRt2/detection_loss‚2.1767650124978046 /loss/FRt4/detection_loss‚2.2309970651354107 /loss/FRt2/assignment_loss‚2.848621386654523 /loss/FRt3/assignment_loss‚1.970722012352525 /loss/FBt4/detection_loss‚2.5109904137524692xfS_[ýÕ /loss/FRt4/assignment_loss‚4.040292457171849 1loss/SRqqt2/assignment_loss‚1.613871421230336 1loss/SRqqt2/detection_loss‚1.7226955384016036 0loss/FBt4/assignment_loss‚0.4066797151616894 0loss/SRqqt1/detection_loss‚2.861761924766359 .loss/FBt1/detection_loss‚2.259941586426326 /loss/FBt3/detection_loss‚0.8060408568081374 %loss/total_loss‚48.71877260474845 trainer/global_step‚6299 /loss/FRt1/assignment_loss‚4.860773298475477 /loss/FBt2/assignment_loss‚0.574811094996312 1loss/FBt3/assignment_loss‚0.16714004234806004 1loss/SRqqt4/detection_loss‚1.8091506523654817 /loss/FBt2/detection_loss‚2.9091262102127073 epoch‚5 1loss/SRqqt1/assignment_loss‚2.610520053477514 2loss/SRqqt4/assignment_loss‚1.8342305287084681 1loss/SRqqt3/detection_loss‚1.5888608396053314 _runtime‚4719.285546093 .loss/FRt1/detection_loss‚3.245668846148032 /loss/FRt3/detection_loss‚1.8250546356042225 % _timestamp‚1.7884121088735106e+09 /loss/FRt2/detection_loss‚2.1767650124978046 /loss/FRt4/detection_loss‚2.2309970651354107 /loss/FRt2/assignment_loss‚2.848621386654523 /loss/FRt3/assignment_loss‚1.970722012352525 /loss/FBt4/detection_loss‚2.5109904137524692 _step‚518 2loss/SRqqt3/assignment_loss‚1.2020070355013013 0loss/FBt1/assignment_loss‚0.6420499769465096†©sDÜþ‚(:… ÒäÔÃçµ™ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.544' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.1088 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!#ÿ‚(:˜ ÒäÔˆÿå› network.recv‚ 1.765788e+06) disk./.usagePercent‚73.67741618067562 disk.sda2.in‚0% disk.nvme0n1.out‚62584.6376953125* proc.memory.percent‚17.568230628967285 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚62584.6376953125 disk.sda3.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 190.76953125# proc.memory.rssMB‚ 2878.37890625 network.sent‚ 1.4579568e+07" disk.dm-0.out‚62584.6376953125 disk.rbd2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 190.76953125 disk.dm-1.out‚ 190.76953125 disk.rbd3.out‚0 disk.sda.in‚0 disk.nvme0n1.in‚ 710.40625% disk./.usageGB‚1097.8467445373535 disk.dm-0.in‚ 710.40625 disk.dm-1.in‚0 disk.rbd2.out‚ 109.751953125 disk.nvme0c0n1.in‚ 710.40625$ memory_percent‚98.47149848937988 cpu‚25.385056216781425 disk.sda1.in‚0 disk.rbd1.out‚2.515625' proc.memory.availableMB‚ 250.4296875qÔ _€‚(:„ áäÔ©™ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.43' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.486 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù¦õ ‚(:• áäÔ×ö±›) disk./.usagePercent‚73.67741618067562 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2878.37890625 disk.nvme0n1.in‚ 710.40625 disk.nvme0c0n1.in‚ 710.40625 disk.dm-1.in‚0 disk.dm-1.out‚ 191.69921875 disk.rbd0.in‚0 disk.rbd2.out‚ 109.876953125" disk.dm-0.out‚62683.8662109375* proc.memory.percent‚17.568230628967285 network.recv‚ 1.772104e+06 disk.rbd0.out‚0 disk.sda.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 710.40625$ proc.memory.availableMB‚249.9375 network.sent‚ 1.4688455e+07 disk.sda2.in‚0 disk.sda.out‚ 191.69921875 disk.drbd1777.out‚0 disk.rbd1.out‚2.515625 cpu‚25.384005965130008% disk./.usageGB‚1097.8467445373535% disk.nvme0n1.out‚62683.8662109375 disk.rbd3.out‚0' disk.nvme0c0n1.out‚62683.8662109375 disk.rbd2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 191.69921875 disk.sda1.out‚0 disk.drbd1777.in‚0$ memory_percent‚98.47450256347656eÞ¬³2‚- _wandbruntime‚4753  _runtime‚4753‰‰³äǃÁ _runtime‚4753.361734971 _step‚519 & lr-AdamW/pg1‚5.489483989724198e-05 & lr-AdamW/pg2‚5.489483989724198e-05 trainer/global_step‚6349 % _timestamp‚1.7884121429495885e+09}û!Ì„Æ & lr-AdamW/pg2‚5.489483989724198e-05 trainer/global_step‚6349 % _timestamp‚1.7884121429495885e+09 _runtime‚4753.361734971 _step‚519 & lr-AdamW/pg1‚5.489483989724198e-05‡žÆ©2…- _wandbruntime‚4753  _runtime‚4753¿À|2†ÙO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884121429503043e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36117271 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9112607785472 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83767695 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚170388057162752 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8931767970722 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚6349 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _step‚520 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48899656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48899654 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 212173543 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83767697 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83767697 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 212193583 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚188073417764352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85016925 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8957754959360 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36117804 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚170388074202112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚179115644802048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚188073230761472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚179115662804992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198896382077440 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚179319825133474 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8931933092126 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128425886 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚179115644802048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8957585959424 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128405848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚4753.362660885 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9112824661504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128405848 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚188073230761472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198896141261312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚188073417764352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85016925 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8957754959360 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36117804 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚170388074202112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚179115644802048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚188073230761472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚179115662804992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198896382077440 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚179319825133474 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8931933092126 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128425886 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚179115644802048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8957585959424 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128405848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚4753.362660885 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9112824661504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128405848 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚188073230761472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198896141261312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884121429503043e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36117271 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9112607785472 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83767695 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚170388057162752 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8931767970722 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚6349 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _step‚520 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48899656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48899654 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 212173543 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83767697 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83767697 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 212193583 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚188101893264384 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128425635 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8933322256878 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8933157135266 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚179142914166784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚179142914166784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9114235129856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚170414017384448 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128425635 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚179142932169728 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212226055 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9114024558592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8959148110848 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85030128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212205989 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48907033 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚188102080280576 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83780354 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189812446424064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83780354 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85029600 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8958979097600 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚179347339641326 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992P²½1ŠèM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚170414017384448 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128425635 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚179142932169728 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212226055 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9114024558592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8959148110848 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85030128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212205989 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48907033 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚188102080280576 VHDeviceStatsMonitor.on_train²[Þ\½_batch_end/allocated_bytes.small_pool.current‚ 169013248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83780354 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189812446424064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83780354 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85029600 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8958979097600 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚179347339641326 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212205989 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚170414000345088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚188101893264384 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128425635 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8933322256878 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8933157135266 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚179142914166784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚179142914166784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9114235129856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0‰X¯3â2‹- _wandbruntime‚4754  _runtime‚4754Ø_ÿ®XŒÒ 1loss/SRqqt4/detection_loss‚1.8203473601076339 %loss/total_loss‚49.91637814773455 trainer/global_step‚6349 /loss/FRt1/assignment_loss‚4.913208434979121 .loss/FBt1/detection_loss‚3.237609041588647 2loss/SRqqt3/assignment_loss‚1.3254417621841033 /loss/FBt2/assignment_loss‚0.661314638538493 0loss/FBt3/assignment_loss‚0.3309815215128437 0loss/FBt4/assignment_loss‚0.5027404298478966 epoch‚5 _runtime‚ 4754.00773111 /loss/FRt2/assignment_loss‚2.898314129114151 1loss/SRqqt2/detection_loss‚1.7741428301748583 /loss/FBt2/detection_loss‚2.8883562948968677 /loss/FBt4/detection_loss‚2.5547117431958517 $ _timestamp‚1.788412143595689e+09 /loss/FRt1/detection_loss‚3.1219836001594863 /loss/FRt3/detection_loss‚2.1268436489626765 .loss/FRt4/detection_loss‚2.614274349477556 _step‚522 2loss/SRqqt1/assignment_loss‚2.5742970009644828 2loss/SRqqt4/assignment_loss‚1.8128443668286005 0loss/FRt3/assignment_loss‚2.4699868919948735 1loss/SRqqt2/assignment_loss‚1.368602015709473 1loss/FBt1/assignment_loss‚0.39800647153606405 /loss/FRt2/detection_loss‚2.1971926802396773 0loss/FRt4/assignment_loss‚3.4441127843326993 1loss/SRqqt1/detection_loss‚2.7135442435741424 1loss/SRqqt3/detection_loss‚1.4324194916213553 /loss/FBt3/detection_loss‚0.7351024161929949¤Žç˜]× 2loss/SRqqt4/assignment_loss‚1.8128443668286005 0loss/FRt3/assignment_loss‚2.4699868919948735 1loss/SRqqt2/assignment_loss‚1.368602015709473 1loss/FBt1/assignment_loss‚0.39800647153606405 /loss/FRt2/detection_loss‚2.1971926802396773 0loss/FRt4/assignment_loss‚3.4441127843326993 1loss/SRqqt1/detection_loss‚2.7135442435741424 1loss/SRqqt3/detection_loss‚1.4324194916213553 /loss/FBt3/detection_loss‚0.7351024161929949 1loss/SRqqt4/detection_loss‚1.8203473601076339 %loss/total_loss‚49.91637814773455 trainer/global_step‚6349 /loss/FRt1/assignment_loss‚4.913208434979121 .loss/FBt1/detection_loss‚3.237609041588647 2loss/SRqqt3/assignment_loss‚1.3254417621841033 /loss/FBt2/assignment_loss‚0.661314638538493 0loss/FBt3/assignment_loss‚0.3309815215128437 0loss/FBt4/assignment_loss‚0.5027404298478966 epoch‚5 _runtime‚ 4754.00773111 /loss/FRt2/assignment_loss‚2.898314129114151 1loss/SRqqt2/detection_loss‚1.7741428301748583 /loss/FBt2/detection_loss‚2.8883562948968677 /loss/FBt4/detection_loss‚2.5547117431958517 $ _timestamp‚1.788412143595689e+09 /loss/FRt1/detection_loss‚3.1219836001594863 /loss/FRt3/detection_loss‚2.1268436489626765 .loss/FRt4/detection_loss‚2.614274349477556 _step‚522 2loss/SRqqt1/assignment_loss‚2.5742970009644828Š­`ñŽŽ‚(:ƒ ðäÔ’ßîš gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚94.33' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.866 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01Ø?‚(: ðäÔȦœ disk.sda.out‚ 192.046875" disk.dm-0.out‚62783.0419921875 disk.drbd1777.in‚0# proc.memory.rssMB‚ 2878.37890625 disk.sda1.in‚0 disk.rbd2.out‚ 109.892578125( proc.memory.availableMB‚ 250.19140625 disk.dm-1.in‚0 disk.rbd3.out‚0# memory_percent‚98.4729528427124% disk./.usageGB‚1097.8468475341797 disk.sda.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚2.515625 disk.dm-1.out‚ 192.046875 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 network.sent‚ 1.4693807e+07) disk./.usagePercent‚73.67742309287905 disk.nvme0n1.in‚ 710.40625 disk.rbd0.in‚0 disk.rbd3.in‚0' disk.nvme0c0n1.out‚62783.0419921875 cpu‚25.3829476150909 proc.cpu.threads‚17% disk.nvme0n1.out‚62783.0419921875 network.recv‚ 1.775713e+06 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.in‚0 disk.sda3.out‚ 192.046875 disk.sda1.out‚0 disk.dm-0.in‚ 710.40625 disk.rbd2.in‚0* proc.memory.percent‚17.568230628967285ÛÊ!ß‘‚(:† ÿäÔÁ‡™ gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.184' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³7ÌY!‘‚(:– ÿäÔÃãÝœ disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda.in‚0$ memory_percent‚98.47345352172852 disk.drbd1777.out‚0 disk.dm-1.out‚ 192.6328125 disk.nvme0c0n1.in‚ 710.40625 disk.sda.out‚ 192.6328125& proc.memory.availableMB‚ 250.109375 disk.sda1.out‚0" disk.dm-0.out‚62783.4873046875 disk.dm-1.in‚0 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 710.40625 disk.rbd3.out‚0 disk.sda3.out‚ 192.6328125 proc.cpu.threads‚17% disk.nvme0n1.out‚62783.4873046875 disk.dm-0.in‚ 710.40625 disk.rbd0.in‚0 network.sent‚ 1.4801727e+07 network.recv‚ 1.781782e+06) disk./.usagePercent‚73.67742309287905% disk./.usageGB‚1097.8468475341797# proc.memory.rssMB‚ 2878.37890625* proc.memory.percent‚17.568230628967285 cpu‚25.381849200967192 disk.sda1.in‚0 disk.rbd1.out‚2.515625 disk.rbd0.out‚0' disk.nvme0c0n1.out‚62783.4873046875 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd2.in‚0! disk.rbd2.out‚109.97900390625PËÍŠ’‚(:… Ž‚äÔÖ¤‘™ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚92.972' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5944 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=Y®“‚(:’ Ž‚äÔ¯¾–› disk.rbd0.in‚0 disk.dm-0.in‚ 710.40625) disk./.usagePercent‚73.67742309287905% disk./.usageGB‚1097.8468475341797 disk.drbd1777.out‚0% disk.nvme0c0n1.out‚62882.71484375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0$ memory_percent‚98.47338199615479( proc.memory.availableMB‚ 250.12109375* proc.memory.percent‚17.568230628967285 disk.sda3.out‚ 192.9765625 disk.sda.out‚ 192.9765625 disk.dm-0.out‚62882.71484375 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 710.40625 network.sent‚ 1.4807885e+07 disk.sda3.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2878.37890625 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 192.9765625 disk.rbd3.in‚0 cpu‚25.380831271356012 network.recv‚ 1.786392e+06 disk.sda.in‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 710.40625! disk.rbd2.out‚109.99462890625 proc.cpu.threads‚17 disk.rbd1.out‚2.515625# disk.nvme0n1.out‚62882.71484375®3Þê2”- _wandbruntime‚4787  _runtime‚4787g7˜áÆ•À % _timestamp‚1.7884121771773138e+09 _runtime‚ 4787.58946214 _step‚523 & lr-AdamW/pg1‚5.416787604995261e-05 & lr-AdamW/pg2‚5.416787604995261e-05 trainer/global_step‚6399Bg6½Ë–Å & lr-AdamW/pg2‚5.416787604995261e-05 trainer/global_step‚6399 % _timestamp‚1.7884121771773138e+09 _runtime‚ 4787.58946214 _step‚523 & lr-AdamW/pg1‚5.416787604995261e-05‹ÚDG 2—- _wandbruntime‚4787  _runtime‚4787­Kq(ß'˜ÙO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 % _timestamp‚1.7884121771780405e+09 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129398550 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85658295 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36389791 _runtime‚4787.590377885 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84400645 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚189506370320896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚537 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84400647 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9183735013888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49268506 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49268504 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129418588 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84400647 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚180686638229446 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129418588 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9027426281984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9001404910534 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9001239789130 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚171685233318912 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚189506557323776 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚180479113038848 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚180479113038848 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚189506370320896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85658834 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36390328 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9027426281984 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84400645 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚171685216279552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚180479131041792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9027257282048 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200412701122048=fÆ!™ÞO E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚189506370320896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85658834 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36390328 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9027426281984 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84400645 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚171685216279552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚180479131041792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9027257282048 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200412701122048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 % _timestamp‚1.7884121771780405e+09 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129398550 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85658295 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36389791 _runtime‚4787.590377885 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84400645 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚189506370320896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚537 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84400647 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9183735013888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49268506 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49268504 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129418588 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84400647 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚180686638229446 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.onIøN[Ô_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129418588 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9027426281984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9001404910534 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9001239789130 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚171685233318912 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚189506557323776 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚180479113038848 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚180479113038848 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064ŒèÝ+22š- _wandbruntime‚4788  _runtime‚4788)e ‘é&›ãM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84413306 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9184943121408 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚180506382403584 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85671573 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚180713788415562 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9002628953674 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49275883 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚180506400406528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚200443267273728 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚180506382403584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 213831641 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4788.242303284 _step‚525 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9002794075286 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129438401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129418337 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200443036956672 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84413306 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129438401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9028819433472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9028819433472 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 213851707 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85671051 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚171711159461888 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚200443267273728 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚180506382403584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 213831641 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4788.242303284 _step‚525 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9002794075286 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129438401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129418337 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200443036956672 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84413306 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129438401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9028819433472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9028819433472 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 213851707 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85671051 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚171711159461888 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚180713788415562 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9002628953674 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49275883 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚180506400406528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408A(2- _wandbruntime‚4788  _runtime‚4788ØO£\žÖ 1loss/SRqqt1/assignment_loss‚2.339912614096766 2loss/SRqqt3/assignment_loss‚1.2875427445024252 1loss/FBt3/assignment_loss‚0.28671979072552595 /loss/FRt1/detection_loss‚2.9650935633429167 1loss/SRqqt2/detection_loss‚1.4632852281842912 0loss/SRqqt3/detection_loss‚1.533442362770438 .loss/FBt2/detection_loss‚3.833605134487152 /loss/FBt4/detection_loss‚3.8912538774311543 0loss/FRt4/assignment_loss‚3.0532842179139457 1loss/FBt2/assignment_loss‚0.45644597121427066 /loss/FRt2/detection_loss‚2.1788126323372126 % _timestamp‚1.7884121778317523e+09 /loss/FRt1/assignment_loss‚4.554440021514893 /loss/FRt4/detection_loss‚2.0957617362340293 trainer/global_step‚6399 1loss/FBt4/assignment_loss‚0.38317305682085134 0loss/SRqqt1/detection_loss‚2.818611698306125 _step‚526 2loss/SRqqt4/assignment_loss‚1.5932548349758364 0loss/FBt1/assignment_loss‚0.5118170687052043 epoch‚5 _runtime‚4788.243793151 0loss/FRt2/assignment_loss‚2.5960152000188828 .loss/FRt3/assignment_loss‚1.81792245388031 2loss/SRqqt2/assignment_loss‚1.5768747862561472 /loss/FRt3/detection_loss‚1.7862870741635561 /loss/FBt3/detection_loss‚0.7874878668542231 1loss/SRqqt4/detection_loss‚1.8462999508065998 /loss/FBt1/detection_loss‚2.5883941729863484 &loss/total_loss‚48.245738058529106¬˜}daŸÛ 0loss/FBt1/assignment_loss‚0.5118170687052043 epoch‚5 _runtime‚4788.243793151 0loss/FRt2/assignment_loss‚2.5960152000188828 .loss/FRt3/assignment_loss‚1.81792245388031 2loss/SRqqt2/assignment_loss‚1.5768747862561472 /loss/FRt3/detection_loss‚1.7862870741635561 /loss/FBt3/detection_loss‚0.7874878668542231 1loss/SRqqt4/detection_loss‚1.8462999508065998 /loss/FBt1/detection_loss‚2.5883941729863484 &loss/total_loss‚48.245738058529106 1loss/SRqqt1/assignment_loss‚2.339912614096766 2loss/SRqqt3/assignment_loss‚1.2875427445024252 1loss/FBt3/assignment_loss‚0.28671979072552595 /loss/FRt1/detection_loss‚2.9650935633429167 1loss/SRqqt2/detection_loss‚1.4632852281842912 0loss/SRqqt3/detection_loss‚1.533442362770438 .loss/FBt2/detection_loss‚3.833605134487152 /loss/FBt4/detection_loss‚3.8912538774311543 0loss/FRt4/assignment_loss‚3.0532842179139457 1loss/FBt2/assignment_loss‚0.45644597121427066 /loss/FRt2/detection_loss‚2.1788126323372126 % _timestamp‚1.7884121778317523e+09 /loss/FRt1/assignment_loss‚4.554440021514893 /loss/FRt4/detection_loss‚2.0957617362340293 trainer/global_step‚6399 1loss/FBt4/assignment_loss‚0.38317305682085134 0loss/SRqqt1/detection_loss‚2.818611698306125 _step‚526 2loss/SRqqt4/assignment_loss‚1.5932548349758364ŽˆcÇ‘ ‚(:† ‚äÔ÷¹¤™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.508' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7016 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 âì ¡‚(:• ‚äÔ´ØÕ› disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 193.50390625 disk.rbd1.in‚0! disk.dm-0.out‚62982.232421875 disk.rbd0.out‚0! disk.rbd2.out‚111.63134765625& disk.nvme0c0n1.out‚62982.232421875 disk.drbd1777.out‚0$ disk.nvme0n1.out‚62982.232421875 network.recv‚ 1.790959e+06) disk./.usagePercent‚73.67743128512015 disk.drbd1777.in‚0 network.sent‚ 1.4814095e+07 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.out‚0# memory_percent‚98.4734058380127 cpu‚25.37979725259347% disk./.usageGB‚1097.8469696044922 disk.nvme0c0n1.in‚ 710.40625 disk.dm-0.in‚ 710.40625 disk.sda3.out‚ 193.50390625 disk.dm-1.out‚ 193.50390625 disk.sda1.in‚0 disk.nvme0n1.in‚ 710.40625 proc.cpu.threads‚17 disk.dm-1.in‚0' proc.memory.availableMB‚ 250.1171875# proc.memory.rssMB‚ 2878.37890625* proc.memory.percent‚17.568230628967285 disk.rbd1.out‚2.515625 disk.rbd0.in‚0 disk.rbd2.in‚0*É)‘¢‚(:† ¬‚äÔñîÇ™ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚111.901' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ií-£‚(:û ¬‚äÔ£˜û›% disk./.usageGB‚1097.8469696044922 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda.in‚0 disk.sda1.in‚0$ memory_percent‚98.47211837768555 disk.drbd1777.in‚0 disk.dm-0.in‚ 710.40625 disk.dm-1.out‚194 disk.rbd2.in‚0& disk.nvme0c0n1.out‚63081.361328125 disk.sda3.in‚0# proc.memory.rssMB‚ 2878.37890625 cpu‚25.378771869562325 network.recv‚ 1.796555e+06 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda3.out‚194 disk.rbd1.out‚2.515625 disk.sda2.out‚0 disk.sda.out‚194& proc.memory.availableMB‚ 250.328125 proc.cpu.threads‚17) disk./.usagePercent‚73.67743128512015$ disk.nvme0n1.out‚63081.361328125! disk.rbd2.out‚111.64697265625 network.sent‚ 1.4921788e+07 disk.rbd1.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 710.40625! disk.dm-0.out‚63081.361328125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0* proc.memory.percent‚17.568230628967285Ê/2¤- _wandbruntime‚4822  _runtime‚4822ðñ Ç¥Á & lr-AdamW/pg1‚5.344015144633274e-05 & lr-AdamW/pg2‚5.344015144633274e-05 trainer/global_step‚6449 % _timestamp‚1.7884122117045643e+09 _runtime‚4822.116726646 _step‚527ŒÀNvÌ¦Æ _runtime‚4822.116726646 _step‚527 & lr-AdamW/pg1‚5.344015144633274e-05 & lr-AdamW/pg2‚5.344015144633274e-05 trainer/global_step‚6449 % _timestamp‚1.7884122117045643e+09p½ e2§- _wandbruntime‚4822  _runtime‚4822Æ_…¨…¨ÙO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 trainer/global_step‚6449 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 215444887 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86299061 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚172982375396352 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 _step‚528 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚190939509880320 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9096928604672 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚181842599278592 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚181842581275648 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9096928604672 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130411290 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚192674828298240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9070876728942 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226136064 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚201929206238208 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚181842599278592 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9070711607538 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408T!I”ä'©ÞO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9097097604608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9254377939968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚182053269164654 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚172982392435712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚190939696883200 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36661705 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚201929432374272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49637354 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚181842581275648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9070876728942 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226136064 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚201929206238208 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚181842599278592 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9070711607538 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 trainer/global_step‚6449 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 215444887 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86299061 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚172982375396352 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 _step‚528 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚190939509880320 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9096928604672 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚181842599278592 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚181842581275648 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9096928604672 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130411290 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚192674828298240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚484 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49644733 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36666745 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86311476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9255799281664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85046254 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9072265893694 trainer/global_step‚6449 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130411039 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚192703765186560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚529 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9098321742848 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228219904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _runtime‚4822.769292563 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 215457293 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86311960 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚201959768748032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚201959540528128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚173008318578688 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚181869850640384 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9256003561472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚190968359399424 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚190968172383232 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85046256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚182080601511742 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚182080419350770 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49644731 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130411039 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85046256 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9098490756096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85046254 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9098490756096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 $ _timestamp‚1.788412212356611e+09 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215477359 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚482 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130431103 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚181869868643328 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130431103 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚181869868643328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9098321742848 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192703741246464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217Td³…¬çM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36667227 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚173008335618048 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228219904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _runtime‚4822.769292563 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 215457293 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86311960 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚201959768748032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚201959540528128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚173008318578688 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚181869850640384 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9256003561472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚190968359399424 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚190968172383232 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85046256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚182080601511742 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚182080419350770 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49644731 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130411039 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85046256 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9098490756096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85046254 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9098490756096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 $ _timestamp‚1.788412212356611e+09 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215477359 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚482 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130431103 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚181869868643328 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130431103 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚181869868643328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9098321742848 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192703741246464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 215457293 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚181869850640384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚484 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.8jÇh current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49644733 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36666745 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86311476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9255799281664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85046254 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9072265893694 trainer/global_step‚6449 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130411039 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚192703765186560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚529 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9098321742848 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131403992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚203445749045760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚203445987764736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚183206049512448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚183206067515392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚192372836442624 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 217050499 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9140183425946 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 trainer/global_step‚6499 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚192372649439744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194120499649536 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131383954 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36930131 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚174279551552512 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 217070539 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131403992 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86935815 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50006206 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131383954 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚183206049512448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86936337 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50006204 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36929611 % _timestamp‚1.7884122459855664e+09 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _step‚532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9325273336320 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚192372649439744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚183419717939098 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 _runtime‚ 4856.39791534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194120475709440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚174279534513152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0z¬Bã'¹ÝO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9325488115200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 217050499 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9140183425946 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 trainer/global_step‚6499 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚192372649439744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194120499649536 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131383954 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36930131 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚174279551552512 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 217070539 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131403992 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86935815 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50006206 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131383954 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚183206049512448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86936337 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50006204 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36929611 % _timestamp‚1.7884122459855664e+09 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _step‚532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9325273336320 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚192372649439744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚183419717939098 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 _runtime‚ 4856.39791534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194120475709440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚174279534513152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9166768927232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85666547 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9166768927232 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131403992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚203445749045760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚203445987764736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚183206049512448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚183206067515392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚192372836442624 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217103011 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131403741 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 217103011 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86948454 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚203476079322112 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚174305477695488 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚183233318877184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 trainer/global_step‚6499 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85679204 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131403741 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194149388657664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚183233336880128 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85679204 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85679206 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9141572590490 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86948454 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚203476079322112 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚174305477695488 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚183233318877184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 trainer/global_step‚6499 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85679204 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131403741 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194149388657664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚183233336880128 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85679204 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85679206 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9141572590490 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217103011 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131403741 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 217103011 P[²Ê×O ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚184569535752192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚514 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195566147060736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9236440249856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37202142 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204962273847808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87577196 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50375054 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132396694 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9396150727168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _step‚536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9209820365758 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86299497 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193805788999168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 t}—â'ËÜO WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 trainer/global_step‚6549 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132376656 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184569517749248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204962504178176 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132396694 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9236271249920 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86299497 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37202656 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132396694 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9396150727168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _step‚536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9209820365758 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86299497 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193805788999168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚184569535752192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚514 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195566147060736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9236440249856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37202142 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204962273847808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87577196 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50375054 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚184596805116928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚184596787113984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚195595060008960 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86312154 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 % _timestamp‚1.7884122812602408e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87589927 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚4891.672931554 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9237833401344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86312154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37207496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218728663 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚537 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87590429 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚502 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132416507 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37207996 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218728663 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9237664388096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚184596787113984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚193834638518272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218708597 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86312156 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132396443 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚175602653851648 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚195595036068864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚175602636812288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193834451502080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚184596805116928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚184596787113984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚195595060008960 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86312154 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 % _timestamp‚1.7884122812602408e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87589927 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚4891.672931554 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9237833401344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86312154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37207496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218728663 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚537 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87590429 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚502 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132416507 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37207996 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218728663 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9237664388096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚184596787113984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚193834638518272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218708597 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86312156 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132396443 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚175602653851648 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚195595036068864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚175602636812288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193834451502080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9306111572480 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9305942572544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚513 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚185932985986048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9305942572544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 133389396 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚176873852746752 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9279127062762 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚186152979809514 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚515 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220321843 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88217723 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37474332 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚206479056351232 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚206478819729408 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚195239115561472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚185933003988992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86932445 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9279292184166 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197011770531840 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50743904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚195238928558592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚185933003988992 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚186153161970278 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚176873869786112 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚4925.971120715 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚195238928558592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚540 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197011794471936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1X‘'¸ä'ÛÞO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚206478819729408 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚195239115561472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚185933003988992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86932445 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9279292184166 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197011770531840 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50743904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚195238928558592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚185933003988992 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚186153161970278 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚176873869786112 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚4925.971120715 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚195238928558592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚540 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197011794471936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86932447 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86932445 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 133389396 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9306111572480 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9305942572544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚513 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚185932985986048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9305942572544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 133389396 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚176873852746752 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9279127062762 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚186152979809514 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚515 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220321843 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88217723 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37474332 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚206479056351232 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064œ\ŸÀ¬2Ü- _wandbruntime‚4926  _runtime‚4926ª>KÏÝãM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚176899812968448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50751281 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9280681348918 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚185960255350784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9307335710720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86945104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50751283 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚206509384486400 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚491 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚186180494317366 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚186180312156394 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220334249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9468677066240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚176899795929088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88231057 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9280516227306 _runtime‚4926.631832064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86945104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9468462300672 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220354315 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚206509145780736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220354315 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed–—†jÌ ‚195267591061504 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133409209 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚185960273353728 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚195267778077696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚493 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88230564 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133389145 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 133389145 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884123162190866e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133409209 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚197040707420160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 trainer/global_step‚6599 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220334249 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37479283 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9307504723968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚185960255350784 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192/×g²î&ÞèM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86945106 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9280681348918 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚185960255350784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9307335710720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86945104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50751283 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚206509384486400 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚491 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚186180494317366 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚186180312156394 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220334249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9468677066240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚176899795929088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88231057 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9280516227306 _runtime‚4926.631832064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86945104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9468462300672 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220354315 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚206509145780736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220354315 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚195267591061504 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133409209 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚185960273353728 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚195267778077696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚493 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88230564 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133389145 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 133389145 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884123162190866e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133409209 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚197040707420160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 trainer/global_step‚6599 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220334249 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37479283 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9307504723968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚185960255350784 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚176899812968448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50751281 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚187296472225792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _step‚544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚207995585093120 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134362060 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9375613895168 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 221927455 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88857389 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51112754 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚178171028902912 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚187296454222848 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚187296454222848 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87565397 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9375613895168 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9538143209984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚196672068118016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚532 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚196672255120896 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚187519610744722 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88857923 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0æhÂZ¤ ëÜO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚198457441883136 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9375782895104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87565395 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚187519610744722 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88857923 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatás,…>sMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 134382098 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134382098 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37744635 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9537938916864 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9348764002574 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87565395 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 $ _timestamp‚1.788412349999493e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51112756 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _runtime‚ 4960.41180541 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚187296472225792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _step‚544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚207995585093120 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134362060 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9375613895168 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 221927455 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88857389 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51112754 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚178171028902912 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚187296454222848 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚187296454222848 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87565397 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9375613895168 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9538143209984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚196672068118016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚532 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚196672255120896 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚178196972085248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚178196955045888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚208025687875584 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9539565458432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚198486354831360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9377007033344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87578056 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87578056 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚196700917637120 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37750591 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 % _timestamp‚1.7884123506459322e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 _runtime‚4961.058742503 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 221959901 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187323741590528 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 134381847 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 trainer/global_step‚6649 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134381847 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9377176046592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚196700917637120 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9377176046592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 221959901 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚196700730620928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚187323741590528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9377007033344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚198486330891264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 221979967 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37750074 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88870205 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9350153167326 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 221979967 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88870724 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9539356984320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87578054 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87578054 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _step‚545 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134401911 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192Cknî&îèM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9377176046592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 221959901 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚196700730620928 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚187323741590528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9377007033344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚198486330891264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 221979967 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37750074 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88870205 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9350153167326 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 221979967 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88870724 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9539356984320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87578054 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87578054 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _step‚545 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134401911 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚208025920289792 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚187323723587584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 134401911 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9349988045714 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚178196972085248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚178196955045888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚208025687875584 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9539565458432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚198486354831360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9377007033344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87578056 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87578056 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚196700917637120 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37750591 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 % _timestamp‚1.7884123506459322e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 _runtime‚4961.058742503 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 221959901 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187323741590528 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 134381847 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 trainer/global_step‚6649 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134381847 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9377176046592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚196700917637120 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217¡ax]2ï- _wandbruntime‚4961  _runtime‚4961S–ZðÔ 0loss/FBt2/assignment_loss‚0.2923230240121484 /loss/FRt4/detection_loss‚2.0776922553777695 .loss/FBt4/detection_loss‚2.703095248767308 0loss/FBt1/assignment_loss‚0.5125253796429509 0loss/FBt4/assignment_loss‚0.5658905910594123 % _timestamp‚1.7884123506482947e+09 2loss/SRqqt4/assignment_loss‚1.8670509381880682 /loss/FRt3/detection_loss‚1.8588307664050894 /loss/FBt3/detection_loss‚0.9148748713957414 _runtime‚4961.060339281 /loss/FRt2/assignment_loss‚3.379651861190796 0loss/SRqqt1/detection_loss‚2.469174432403901 &loss/total_loss‚50.426687892601876 /loss/FRt2/detection_loss‚2.2393543738126755 /loss/FBt1/detection_loss‚2.2645443439483643 0loss/FBt3/assignment_loss‚0.2007756254376499 1loss/SRqqt4/detection_loss‚1.6802345644081793 epoch‚5 trainer/global_step‚6649 /loss/FRt1/assignment_loss‚5.142354575070468 1loss/SRqqt1/assignment_loss‚2.487127318101771 2loss/SRqqt2/assignment_loss‚1.6690381069493014 2loss/SRqqt3/assignment_loss‚1.4466276237222015 0loss/SRqqt3/detection_loss‚1.476120585056602 /loss/FBt2/detection_loss‚4.3377155264218645 _step‚546 0loss/FRt3/assignment_loss‚2.6304590681851923 0loss/FRt4/assignment_loss‚3.5234422500316915 /loss/FRt1/detection_loss‚2.8813711603482566 1loss/SRqqt2/detection_loss‚1.8064134026644751êÕ9ó_ñÙ % _timestamp‚1.7884123506482947e+09 2loss/SRqqt4/assignment_loss‚1.8670509381880682 /loss/FRt3/detection_loss‚1.8588307664050894 /loss/FBt3/detection_loss‚0.9148748713957414 _runtime‚4961.060339281 /loss/FRt2/assignment_loss‚3.379651861190796 0loss/SRqqt1/detection_loss‚2.469174432403901 &loss/total_loss‚50.426687892601876 /loss/FRt2/detection_loss‚2.2393543738126755 /loss/FBt1/detection_loss‚2.2645443439483643 0loss/FBt3/assignment_loss‚0.2007756254376499 1loss/SRqqt4/detection_loss‚1.6802345644081793 epoch‚5 trainer/global_step‚6649 /loss/FRt1/assignment_loss‚5.142354575070468 1loss/SRqqt1/assignment_loss‚2.487127318101771 2loss/SRqqt2/assignment_loss‚1.6690381069493014 2loss/SRqqt3/assignment_loss‚1.4466276237222015 0loss/SRqqt3/detection_loss‚1.476120585056602 /loss/FBt2/detection_loss‚4.3377155264218645 _step‚546 0loss/FRt3/assignment_loss‚2.6304590681851923 0loss/FRt4/assignment_loss‚3.5234422500316915 /loss/FRt1/detection_loss‚2.8813711603482566 1loss/SRqqt2/detection_loss‚1.8064134026644751 0loss/FBt2/assignment_loss‚0.2923230240121484 /loss/FRt4/detection_loss‚2.0776922553777695 .loss/FBt4/detection_loss‚2.703095248767308 0loss/FBt1/assignment_loss‚0.5125253796429509 0loss/FBt4/assignment_loss‚0.5658905910594123¢¨‹cl‘ò‚(:† ƒäÔï´Â™ gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚100.603' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.1206 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èÚ¥Ä(ó‚(: ƒäԱΞœ network.recv‚ 1.845303e+06 disk.sda2.out‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2878.37890625 proc.cpu.threads‚17% disk./.usageGB‚1096.7856407165527 disk.sda1.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.in‚0 disk.drbd1777.in‚0% disk.nvme0n1.out‚63548.9169921875* proc.memory.percent‚17.568230628967285 disk.sda1.out‚0! disk.rbd2.out‚115.41748046875 network.sent‚ 1.5388825e+07 disk.rbd1.out‚ 2.822265625 disk.dm-0.in‚ 710.40625 disk.dm-1.out‚ 199.70703125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 199.70703125 cpu‚25.368835181326524) disk./.usagePercent‚73.60620461293627 disk.rbd0.out‚0' disk.nvme0c0n1.out‚63548.9169921875 disk.sda3.out‚ 199.70703125$ memory_percent‚98.47350120544434 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚ 710.40625" disk.dm-0.out‚63548.9169921875' proc.memory.availableMB‚ 250.1015625þbÊ'œô‚(:‘ уäÔ¼û˜™ gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.429' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.085800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gØĸ,õ‚(:š уäÔ‰ðÙ› disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 710.40625' disk.nvme0c0n1.out‚63550.2802734375 disk.sda1.in‚0% disk.nvme0n1.out‚63550.2802734375% disk./.usageGB‚1096.7847366333008 disk.sda3.in‚0 disk.drbd1777.out‚0 cpu‚25.367935182454517 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 200.3359375" disk.dm-0.out‚63550.2802734375* proc.memory.percent‚17.568230628967285 network.recv‚ 1.851424e+06) disk./.usagePercent‚73.60614393915066 disk.sda1.out‚0 disk.sda.out‚ 2òÖFþù00.3359375 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd1.out‚ 2.822265625 disk.nvme0n1.in‚ 710.40625 disk.dm-0.in‚ 710.40625( proc.memory.availableMB‚ 249.76953125# proc.memory.rssMB‚ 2878.37890625 disk.rbd0.out‚0 disk.sda.in‚0 disk.drbd1777.in‚0 network.sent‚ 1.5497417e+07! disk.rbd2.out‚115.47998046875 disk.rbd3.out‚0 disk.sda2.out‚0# memory_percent‚98.4755277633667 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 200.3359375´a¿q‘ö‚(:† àƒäÔܸ”™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚111.056' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2112 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`™o²"÷‚(:— àƒäÔ‚»› cpu‚25.36693788015649' disk.nvme0c0n1.out‚63550.6474609375 disk.sda.out‚ 200.76171875 disk.rbd2.in‚0 disk.sda.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0# proc.memory.availableMB‚249.625# proc.memory.rssMB‚ 2878.37890625* proc.memory.percent‚17.568230628967285 network.sent‚ 1.5502776e+07 network.recv‚ 1.85529e+06 disk.nvme0n1.in‚ 710.40625 disk.dm-0.in‚ 710.40625 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.out‚0! disk.rbd2.out‚115.49560546875 disk.rbd3.out‚0 disk.sda1.in‚0) disk./.usagePercent‚73.60620922107188% disk.nvme0n1.out‚63550.6474609375" disk.dm-0.out‚63550.6474609375 disk.dm-1.out‚ 200.76171875 disk.sda3.out‚ 200.76171875 disk.sda1.out‚0$ memory_percent‚98.47640991210938 disk.rbd3.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0% disk./.usageGB‚1096.7857093811035 disk.rbd1.in‚0 disk.rbd1.out‚ 2.822265625 disk.rbd0.in‚0FÙ…2ø- _wandbruntime‚4995  _runtime‚4995ÉkÈù _step‚547 ' lr-AdamW/pg1‚4.9795428846705414e-05 ' lr-AdamW/pg2‚4.9795428846705414e-05 trainer/global_step‚6699 % _timestamp‚1.7884123848456333e+09 _runtime‚ 4995.25775829pi¥ÍúÇ ' lr-AdamW/pg1‚4.9795428846705414e-05 ' lr-AdamW/pg2‚4.9795428846705414e-05 trainer/global_step‚6699 % _timestamp‚1.7884123848456333e+09 _runtime‚ 4995.25775829 _step‚547£2®×2û- _wandbruntime‚4995  _runtime‚4995ü¤ÌÞ'üØO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9445285217792 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88198345 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚ 4995.25875616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51481606 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚198105207677440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚188659922459648 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚198105207677440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚198105394680320 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51481604 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38017046 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 trainer/global_step‚6699 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88198347 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚188659940462592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9445454217728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209511910843392 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚188886241679930 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9418070699578 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 223553107 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135354762 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚199903065354240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9418235820982 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚89498652 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚557 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 % _timestamp‚1.7884123848463845e+09 _step‚548 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135374800 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚555 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚188886423840694 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135354762 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 223573147 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9608845489152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚179468188019712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209512143270912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚188659940462592ÓNÅÛã'ýÝO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209511910843392 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚188886241679930 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9418070699578 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 223553107 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135354762 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚199903065354240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9418235820982 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚89498652 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚557 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 % _timestamp‚1.7884123848463845e+09 _step‚548 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135374800 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚555 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚188886423840694 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135354762 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 223573147 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9608845489152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚179468188019712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209512143270912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚188659940462592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9445285217792 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88198345 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚ 4995.25875616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51481606 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚198105207677440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚188659922459648 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚198105207677440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚198105394680320 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51481604 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38017046 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 trainer/global_step‚6699 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88198347 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚188659940462592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9445454217728¤®7·2þ- _wandbruntime‚4995  _runtime‚4995’Õaß&ÿãM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚198134057196544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 223585553 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9446847369216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9610260869632 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9446847369216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88211004 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚179494114162688 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 223605619 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚188913756187782 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚188687191824384 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 223605619 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51488983 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚209542239172096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88211004 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚198134057196544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51488981 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188913574026810 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚89511347 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9446678355968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚179494131202048 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚188687191824384 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89510809 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚199932002242560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88211006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209542477877760 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9419459864122 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚198133870180352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚188687209827328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9446678355968 % _timestamp‚1.7884123854982946e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88211006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 135394613 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38021828 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135374549 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 I?DeviceStatsMonitor.on_train_batchô…YˆÎ_end/active.small_pool.current‚20064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 223585553 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9419624985734. êFî&€èM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 223585553 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9419624985734 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚198134057196544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 223585553 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9446847369216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9610260869632 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9446847369216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88211004 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚179494114162688 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 223605619 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚188913756187782 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚188687191824384 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 223605619 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51488983 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚209542239172096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88211004 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚198134057196544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51488981 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188913574026810 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚89511347 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9446678355968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚179494131202048 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚188687191824384 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89510809 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚199932002242560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88211006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209542477877760 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9419459864122 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚198133870180352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚188687209827328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9446678355968 % _timestamp‚1.7884123854982946e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88211006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 135394613 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38021828 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135374549 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217¥®ò~2- _wandbruntime‚4995  _runtime‚4995x-Z‚Ô /loss/FRt1/assignment_loss‚4.698338994613061 1loss/SRqqt1/assignment_loss‚2.314305814412924 2loss/SRqqt3/assignment_loss‚1.4464524151971105 1loss/FBt2/assignment_loss‚0.39496147342854077 /loss/FRt4/detection_loss‚2.2434853675230495 epoch‚5 % _timestamp‚1.7884123855005133e+09 _step‚550 0loss/FRt4/assignment_loss‚3.7221034619043456 /loss/SRqqt3/detection_loss‚1.33818832760463 0loss/SRqqt4/detection_loss‚1.858181194961071 /loss/FBt3/detection_loss‚0.7552046449756888 0loss/FRt2/assignment_loss‚2.7888684807991493 2loss/SRqqt4/assignment_loss‚1.2068412280455232 1loss/FBt3/assignment_loss‚0.23990124006446054 .loss/FRt2/detection_loss‚2.227088603438163 /loss/FBt4/detection_loss‚2.7740931063890457 1loss/FBt1/assignment_loss‚0.39916963732824884 1loss/SRqqt1/detection_loss‚2.9170909340565023 /loss/FBt1/detection_loss‚2.9054648005045376 /loss/FRt3/detection_loss‚1.7559648078794663 0loss/SRqqt2/detection_loss‚1.360716135083855 .loss/FBt2/detection_loss‚2.558804604742262 _runtime‚4995.912556349 2loss/SRqqt2/assignment_loss‚1.4606634164591887 0loss/FBt4/assignment_loss‚0.5046736934697643 trainer/global_step‚6699 0loss/FRt3/assignment_loss‚1.7331779908675413 /loss/FRt1/detection_loss‚3.1816556545404286 %loss/total_loss‚46.78539602828856±?_ƒÙ 2loss/SRqqt2/assignment_loss‚1.4606634164591887 0loss/FBt4/assignment_loss‚0.5046736934697643 trainer/global_step‚6699 0loss/FRt3/assignment_loss‚1.7331779908675413 /loss/FRt1/detection_loss‚3.1816556545404286 %loss/total_loss‚46.78539602828856 /loss/FRt1/assignment_loss‚4.698338994613061 1loss/SRqqt1/assignment_loss‚2.314305814412924 2loss/SRqqt3/assignment_loss‚1.4464524151971105 1loss/FBt2/assignment_loss‚0.39496147342854077 /loss/FRt4/detection_loss‚2.2434853675230495 epoch‚5 % _timestamp‚1.7884123855005133e+09 _step‚550 0loss/FRt4/assignment_loss‚3.7221034619043456 /loss/SRqqt3/detection_loss‚1.33818832760463 0loss/SRqqt4/detection_loss‚1.858181194961071 /loss/FBt3/detection_loss‚0.7552046449756888 0loss/FRt2/assignment_loss‚2.7888684807991493 2loss/SRqqt4/assignment_loss‚1.2068412280455232 1loss/FBt3/assignment_loss‚0.23990124006446054 .loss/FRt2/detection_loss‚2.227088603438163 /loss/FBt4/detection_loss‚2.7740931063890457 1loss/FBt1/assignment_loss‚0.39916963732824884 1loss/SRqqt1/detection_loss‚2.9170909340565023 /loss/FBt1/detection_loss‚2.9054648005045376 /loss/FRt3/detection_loss‚1.7559648078794663 0loss/SRqqt2/detection_loss‚1.360716135083855 .loss/FBt2/detection_loss‚2.558804604742262 _runtime‚4995.912556349¦Æ𛄂(: ïƒäԋ쨙 gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.526' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.705199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m­–…‚(: ïƒäÔ„øÒ› disk.dm-1.in‚0 disk.dm-1.out‚ 201.48828125 disk.rbd3.in‚0 network.sent‚ 1.5508206e+07 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 115.64453125 disk.nvme0c0n1.in‚ 710.40625% disk.nvme0c0n1.out‚63552.87890625 disk.sda2.in‚0 disk.drbd1777.in‚0# disk.nvme0n1.out‚63552.87890625 disk.dm-0.out‚63552.87890625 disk.sda.out‚ 201.48828125* proc.memory.percent‚17.568230628967285% disk./.usageGB‚1096.6164207458496 disk.rbd2.in‚0 disk.sda2.out‚0% proc.memory.availableMB‚ 249.34375 disk.sda1.in‚0 disk.nvme0n1.in‚ 710.40625 disk.rbd3.out‚0 disk.sda3.in‚0 cpu‚25.36599159275834 proc.cpu.threads‚17 network.recv‚ 1.858748e+06 disk.rbd1.in‚0 disk.sda3.out‚ 201.48828125 disk.drbd1777.out‚0# memory_percent‚98.4781265258789# proc.memory.rssMB‚ 2878.37890625 disk.rbd1.out‚ 2.822265625 disk.sda1.out‚0 disk.sda.in‚0) disk./.usagePercent‚73.59484811871737 disk.dm-0.in‚ 710.40625bNìµ›†‚(: þƒäÔöºÔ™ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚89.496' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.899199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\tÍ«‡‚(: þƒäԚט disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.568230628967285# proc.memory.rssMB‚ 2878.37890625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 710.40625$ disk.nvme0c0n1.out‚ 63553.2734375 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 2.822265625 disk.nvme0n1.in‚ 710.40625$ memory_percent‚98.47660064697266 cpu‚25.36503693374258 disk.rbd1.in‚0 network.recv‚ 1.865377e+06( disk./.usagePercent‚73.5947362434249% disk./.usageGB‚1096.6147537231445 disk.rbd2.out‚ 115.64453125 disk.sda.out‚ 201.80078125 disk.sda1.out‚0% proc.memory.availableMB‚ 249.59375 disk.dm-0.in‚ 710.40625 disk.dm-0.out‚ 63553.2734375 disk.dm-1.out‚ 201.80078125 disk.drbd1777.in‚0 disk.sda.in‚0" disk.nvme0n1.out‚ 63553.2734375 proc.cpu.threads‚17 disk.sda3.out‚ 201.80078125 network.sent‚ 1.5616869e+07 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0Ñ µ2ˆ- _wandbruntime‚5029  _runtime‚5029…@ ljÁ & lr-AdamW/pg1‚4.906632783141987e-05 & lr-AdamW/pg2‚4.906632783141987e-05 trainer/global_step‚6749 % _timestamp‚1.7884124195078735e+09 _runtime‚5029.920000034 _step‚551µfmÌŠÆ % _timestamp‚1.7884124195078735e+09 _runtime‚5029.920000034 _step‚551 & lr-AdamW/pg1‚4.906632783141987e-05 & lr-AdamW/pg2‚4.906632783141987e-05 trainer/global_step‚6749§·æ9ÿ2‹- _wandbruntime‚5029  _runtime‚5029ý!ü?ß'ŒÙO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 _step‚552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51850456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9514956540416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199538347236864 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚190023408699392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9515125540352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201348736705536 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9514956540416 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 225178759 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199538347236864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88831295 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 225198799 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9679702922240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚211028643920896 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚190023390696448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚190252872615138 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136367502 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9679907215360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚180765347136512 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136347464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38288423 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88831295 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚190023390696448 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90138328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51850454 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚211028415687680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9487542517986^鎱ÞO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199538347236864 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚190023408699392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9515125540352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201348736705536 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9514956540416 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 225178759 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199538347236864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88831295 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 225198799 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9679702922240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚211028643920896 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚190023390696448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚190252872615138 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136367502 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9679907215360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚180765347136512 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136347464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38288423 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88831295 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚190023390696448 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90138328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51850454 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚211028415687680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9487542517986 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 _step‚552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51850456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9514956540416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚190280204962018 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9488931682530 % _timestamp‚1.7884124201467733e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136367251 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225231271 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38293576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90151918 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136367251 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚211058977654784 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136387315 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚180791290318848 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚190050678064128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚199567196755968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90151407 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚201377649653760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚190050660061184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51857831 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚190050660061184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚211058749434880 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚190050678064128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38294085 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 225231271 _step‚553 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88843956 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88843954 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9516349678592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51857833 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚199567196755968 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136387315 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88843956 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 225211205 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228219904 _runtime‚5030.559538656 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225231271 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38293576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90151918 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136367251 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚211058977654784 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136387315 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚180791290318848 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚190050678064128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚199567196755968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90151407 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚201377649653760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚190050660061184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51857831 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚190050660061184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚211058749434880 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚190050678064128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38294085 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 225231271 _step‚553 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88843956 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88843954 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9516349678592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51857833 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚199567196755968 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136387315 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88843956 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 225211205 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228219904 _runtime‚5030.559538656 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚190280204962018 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9488931682530 % _timestamp‚1.7884124201467733e+09©<è2‘- _wandbruntime‚5030  _runtime‚5030§‡¨pW’Ñ 0loss/FRt4/assignment_loss‚3.6127407925744213 1loss/SRqqt3/assignment_loss‚1.568217111094033 1loss/SRqqt4/assignment_loss‚1.705776195594808 /loss/FRt1/detection_loss‚3.2327988545099893 /loss/FRt3/assignment_loss‚2.278035890439461 0loss/FBt3/assignment_loss‚0.3277663754283253 /loss/FRt2/detection_loss‚2.1497088267847344 1loss/SRqqt4/detection_loss‚1.8396943521996338 /loss/FBt1/detection_loss‚2.3664114694846305 /loss/FBt3/detection_loss‚0.7679688904912304 1loss/SRqqt1/detection_loss‚2.5804878723236824 1loss/SRqqt2/detection_loss‚1.6505546948174015 epoch‚5 /loss/FRt4/detection_loss‚1.8299610374435302 2loss/SRqqt1/assignment_loss‚2.1386751032644704 /loss/FBt2/assignment_loss‚0.227888207821007 /loss/FRt3/detection_loss‚2.0049418015726683 1loss/SRqqt3/detection_loss‚1.6137568429112434 _step‚554 0loss/FRt2/assignment_loss‚2.7762668353539928 0loss/FBt1/assignment_loss‚0.6875918807073551 .loss/FBt2/detection_loss‚3.211392778616685 # _timestamp‚1.78841242014887e+09 _runtime‚5030.560908386 2loss/SRqqt2/assignment_loss‚1.2720338485087268 .loss/FBt4/detection_loss‚2.187441340514592 /loss/FRt1/assignment_loss‚5.331866572300593 0loss/FBt4/assignment_loss‚0.6259497425168904 &loss/total_loss‚47.987927317274114 trainer/global_step‚6749 –\“Ö 0loss/FBt3/assignment_loss‚0.3277663754283253 /loss/FRt2/detection_loss‚2.1497088267847344 1loss/SRqqt4/detection_loss‚1.8396943521996338 /loss/FBt1/detection_loss‚2.3664114694846305 /loss/FBt3/detection_loss‚0.7679688904912304 1loss/SRqqt1/detection_loss‚2.5804878723236824 1loss/SRqqt2/detection_loss‚1.6505546948174015 epoch‚5 /loss/FRt4/detection_loss‚1.8299610374435302 2loss/SRqqt1/assignment_loss‚2.1386751032644704 /loss/FBt2/assignment_loss‚0.227888207821007 /loss/FRt3/detection_loss‚2.0049418015726683 1loss/SRqqt3/detection_loss‚1.6137568429112434 _step‚554 0loss/FRt2/assignment_loss‚2.7762668353539928 0loss/FBt1/assignment_loss‚0.6875918807073551 .loss/FBt2/detection_loss‚3.211392778616685 # _timestamp‚1.78841242014887e+09 _runtime‚5030.560908386 2loss/SRqqt2/assignment_loss‚1.2720338485087268 .loss/FBt4/detection_loss‚2.187441340514592 /loss/FRt1/assignment_loss‚5.331866572300593 0loss/FBt4/assignment_loss‚0.6259497425168904 &loss/total_loss‚47.987927317274114 trainer/global_step‚6749 0loss/FRt4/assignment_loss‚3.6127407925744213 1loss/SRqqt3/assignment_loss‚1.568217111094033 1loss/SRqqt4/assignment_loss‚1.705776195594808 /loss/FRt1/detection_loss‚3.2327988545099893 /loss/FRt3/assignment_loss‚2.278035890439461ª%>0X›”‚(: „äԛŬš gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚111.21' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.241999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G1ÚÊ•‚(:’ „äÔ—´Ç› disk.dm-1.out‚ 202.48828125 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 202.48828125 proc.cpu.threads‚17 disk.sda1.out‚0 cpu‚25.364120873425748) disk./.usagePercent‚73.60355493495689 disk.rbd3.in‚0 network.recv‚ 1.870168e+06 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd2.out‚ 117.359375% disk.nvme0c0n1.out‚63686.98828125 disk.dm-0.out‚63686.98828125 disk.sda2.out‚0' proc.memory.availableMB‚ 249.5546875* proc.memory.percent‚17.568230628967285 disk.rbd0.out‚0 disk.sda3.in‚0$ memory_percent‚98.47683906555176 network.sent‚ 1.5623004e+07 disk.sda.in‚0 disk.sda.out‚ 202.48828125 disk.drbd1777.in‚0 disk.rbd1.out‚ 2.822265625 disk.nvme0n1.in‚ 710.40625 disk.dm-0.in‚ 710.40625 disk.dm-1.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.in‚0# proc.memory.rssMB‚ 2878.37890625% disk./.usageGB‚1096.7461585998535# disk.nvme0n1.out‚63686.98828125pQ+–‚(:„ œ„äԣИ™ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.26' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.652 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)´c—‚(: œ„äÔëæÆœ disk.rbd3.out‚0$ memory_percent‚98.47686290740967# proc.memory.rssMB‚ 2878.37890625* proc.memory.percent‚17.568230628967285) disk./.usagePercent‚73.61227915970872% disk./.usageGB‚1096.8761558532715 disk.drbd1777.out‚0 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.822265625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.out‚ 63821.8046875 disk.sda2.out‚0 disk.sda3.out‚ 202.9453125 disk.nvme0n1.in‚ 710.40625 disk.rbd2.out‚ 117.359375( proc.memory.availableMB‚ 249.55078125 cpu‚25.363183000430652 disk.sda.in‚0 disk.sda.out‚ 202.9453125 disk.nvme0c0n1.in‚ 710.40625 disk.dm-0.out‚ 63821.8046875 disk.dm-1.out‚ 202.9453125 network.sent‚ 1.5731078e+07 network.recv‚ 1.876223e+06 disk.dm-0.in‚ 710.40625 disk.sda2.in‚0 disk.sda1.in‚0" disk.nvme0n1.out‚ 63821.8046875 disk.rbd3.in‚0Áý 2˜- _wandbruntime‚5065  _runtime‚5065ÚyîýÆ™À $ _timestamp‚1.788412454824374e+09 _runtime‚5065.236504434 _step‚555 & lr-AdamW/pg1‚4.833752921480617e-05 & lr-AdamW/pg2‚4.833752921480617e-05 trainer/global_step‚6799bŽ©i˚Š$ _timestamp‚1.788412454824374e+09 _runtime‚5065.236504434 _step‚555 & lr-AdamW/pg1‚4.833752921480617e-05 & lr-AdamW/pg2‚4.833752921480617e-05 trainer/global_step‚6799«µŠ“Ë2›- _wandbruntime‚5065  _runtime‚5065i$*â œÙO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226824451 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89464245 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚191619503550346 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9584627863040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137340166 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52219304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9750614535168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89464247 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚212545209236480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 226804411 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 % _timestamp‚1.7884124548251047e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137360204 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚191386876936192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9584627863040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9750825119744 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9557014336394 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89464245 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚202794384116736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚6799 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89464247 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚513 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚191619685711110 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137340166 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚191386876936192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38559493 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚200971486796288 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9584796862976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚511 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚212544974711808 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚182062489213952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚200971486796288 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9584796862976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚511 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚212544974711808 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚182062489213952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9584627863040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137340166 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52219304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9750614535168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89464247 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚212545209236480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 226804411 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 % _timestamp‚1.7884124548251047e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137360204 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚191386876936192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9584627863040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9750825119744 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9557014336394 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89464245 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚202794384116736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚6799 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89464247 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚513 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚191619685711110 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137340166 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚191386876936192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38559493 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90791493 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52226681 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137380017 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚201000336315392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38564812 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚504 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _runtime‚5065.918822441 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚191414128297984 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226856923 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚212575337015808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚191647018058198 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9586021001216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52226683 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚201000336315392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9586021001216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89476906 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137359953 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚212575571527168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚182088449435648 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89476904 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚202823273124864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89476906 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9586190014464 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9558568622550 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚191414146300928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚182088432396288 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137359953 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚504 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _runtime‚5065.918822441 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚191414128297984 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226856923 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚212575337015808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚191647018058198 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9586021001216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52226683 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚201000336315392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9586021001216 TEDeviceStatsMonitor.on_øœ{xÔtrain_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89476906 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137359953 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚212575571527168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚182088449435648 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89476904 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚202823273124864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89476906 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9586190014464 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9558568622550 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚191414146300928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚182088432396288 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137359953 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90791493 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52226681 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137380017 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚201000336315392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38564812 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BÌ¬Æ & lr-AdamW/pg1‚4.760918487740365e-05 & lr-AdamW/pg2‚4.760918487740365e-05 trainer/global_step‚6849 % _timestamp‚1.7884124896494699e+09 _runtime‚5100.061576772 _step‚559¯¼Ž}Á2­- _wandbruntime‚5100  _runtime‚5100#à)óß'®ÙO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚183359648330752 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚202404813358592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚214061781612032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 138332868 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90097197 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9626651276206 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚202404813358592 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 trainer/global_step‚6849 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚192750345172992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 228430063 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚202404626355712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9821535305216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 % _timestamp‚1.7884124896501522e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38832679 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38832136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚192986316646318 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90097197 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52588156 _runtime‚5100.062491205 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9654299185664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9654468185600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90097195 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚192750327170048 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91420290 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚214061781612032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 138332868 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90097197 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9626651276206 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚202404813358592 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 trainer/global_step‚6849 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚192750345172992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 228430063 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚202404626355712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9821535305216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 % _timestamp‚1.7884124896501522e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38832679 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38832136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚192986316646318 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90097197 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52588156 _runtime‚5100.062491205 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9654299185664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9654468185600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90097195 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚192750327170048 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91420290 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 228462509 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138352655 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91433001 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚192777614537728 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚192777614537728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚202433475874816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228482575 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884124902915525e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9655861337088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52595531 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9822959128576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9823161311232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90109856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226122752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚183385608552448 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9655692323840 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202182656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9655861337088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9628040440958 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _runtime‚5100.704427599 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52595533 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚204268920536064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138352655 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚192777596534784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9627875319346 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚202433288858624 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚192777596534784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9627875319346 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚202433288858624 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 228462509 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138352655 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91433001 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚192777614537728 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚192777614537728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚202433475874816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228482575 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884124902915525e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9655861337088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52595531 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9822959128576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9823161311232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90109856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226122752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚183385608552448 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9655692323840 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202182656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9655861337088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9628040440958 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _runtime‚5100.704427599±7Ñm2³- _wandbruntime‚5100  _runtime‚5100¹ÿLŸ[´Õ .loss/FRt4/detection_loss‚2.049890986428811 /loss/FBt3/detection_loss‚0.7478010947688496 _step‚562 2loss/SRqqt4/assignment_loss‚1.6673748901774805 0loss/FBt2/assignment_loss‚0.5389608743296574 _runtime‚5100.705729412 /loss/FRt1/assignment_loss‚4.621980811610366 1loss/FBt4/assignment_loss‚0.46340394261023904 1loss/SRqqt2/detection_loss‚1.7469631159891847 1loss/SRqqt1/assignment_loss‚2.443800926208496 trainer/global_step‚6849 % _timestamp‚1.7884124902933784e+09 2loss/SRqqt2/assignment_loss‚1.2335834468363738 0loss/SRqqt1/detection_loss‚2.799983593924292 /loss/FBt4/detection_loss‚2.2028700560331345 2loss/SRqqt3/assignment_loss‚1.2214292213320732 0loss/FBt1/assignment_loss‚0.2663659831911597 /loss/FRt2/detection_loss‚2.4391326531767845 1loss/SRqqt4/detection_loss‚1.5462527851904593 /loss/FBt2/detection_loss‚2.8548572924402027 0loss/FRt2/assignment_loss‚2.6329648991425834 /loss/FRt4/assignment_loss‚2.785817469541843 1loss/SRqqt3/detection_loss‚1.4424508918821812 /loss/FBt1/detection_loss‚3.3959373831748962 epoch‚5 0loss/FBt3/assignment_loss‚0.1920484274115265 %loss/total_loss‚45.60951820160349 0loss/FRt3/assignment_loss‚1.5466829708644323 /loss/FRt1/detection_loss‚3.0000032096198113 /loss/FRt3/detection_loss‚1.7689612757186501P$ ã`µÚ /loss/FRt1/detection_loss‚3.0000032096198113 /loss/FRt3/detection_loss‚1.7689612757186501 .loss/FRt4/detection_loss‚2.049890986428811 /loss/FBt3/detection_loss‚0.7478010947688496 _step‚562 2loss/SRqqt4/assignment_loss‚1.6673748901774805 0loss/FBt2/assignment_loss‚0.5389608743296574 _runtime‚5100.705729412 /loss/FRt1/assignment_loss‚4.621980811610366 1loss/FBt4/assignment_loss‚0.46340394261023904 1loss/SRqqt2/detection_loss‚1.7469631159891847 1loss/SRqqt1/assignment_loss‚2.443800926208496 trainer/global_step‚6849 % _timestamp‚1.7884124902933784e+09 2loss/SRqqt2/assignment_loss‚1.2335834468363738 0loss/SRqqt1/detection_loss‚2.799983593924292 /loss/FBt4/detection_loss‚2.2028700560331345 2loss/SRqqt3/assignment_loss‚1.2214292213320732 0loss/FBt1/assignment_loss‚0.2663659831911597 /loss/FRt2/detection_loss‚2.4391326531767845 1loss/SRqqt4/detection_loss‚1.5462527851904593 /loss/FBt2/detection_loss‚2.8548572924402027 0loss/FRt2/assignment_loss‚2.6329648991425834 /loss/FRt4/assignment_loss‚2.785817469541843 1loss/SRqqt3/detection_loss‚1.4424508918821812 /loss/FBt1/detection_loss‚3.3959373831748962 epoch‚5 0loss/FBt3/assignment_loss‚0.1920484274115265 %loss/total_loss‚45.60951820160349 0loss/FRt3/assignment_loss‚1.5466829708644323²ïߎ8¶‚(:„ Ø„äÔØÄ…™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.29' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.058 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”ŸB*·‚(:Ÿ Ø„äÔ©Ôî›) disk./.usagePercent‚73.61282368773405 disk.sda2.in‚0 disk.sda.in‚0! disk.rbd2.out‚117.66064453125* proc.memory.percent‚17.568230628967285 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.sda1.in‚0& disk.nvme0n1.out‚63832.81005859375( disk.nvme0c0n1.out‚63832.81005859375 disk.sda3.in‚0& proc.memory.availableMB‚ 249.234375 disk.rbd3.out‚0 disk.rbd1.out‚ 2.822265625# disk.dm-0.out‚63832.81005859375$ memory_percent‚98.47879409790039% disk./.usageGB‚1096.8842697143555 disk.sda3.out‚ 205.05859375 disk.sda.out‚ 205.05859375 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 710.40625 disk.dm-1.in‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 2878.37890625 network.sent‚ 1.5856666e+07 disk.rbd1.in‚0 disk.dm-0.in‚ 710.40625 disk.dm-1.out‚ 205.05859375 cpu‚25.359485242746683 network.recv‚ 1.895095e+06 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.out‚0»ë[ž¸‚(:… ç„äÔÕÔš™ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚99.716' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.9432 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õúõ7'¹‚(:œ ç„äÔ€¹‡œ disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 205.8046875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 710.40625( disk.nvme0c0n1.out‚63833.71630859375 proc.cpu.threads‚17 network.recv‚ 1.900664e+06) disk./.usagePercent‚73.61282727183954 disk.sda2.in‚0 disk.rbd3.in‚0$ memory_percent‚98.47745895385742 cpu‚25.358582663317403 disk.nvme0n1.in‚ 710.40625& proc.memory.availableMB‚ 249.453125 disk.sda3.in‚0# disk.dm-0.out‚63833.71630859375 disk.rbd0.out‚0 network.sent‚ 1.5964523e+07 disk.drbd1777.out‚0 disk.rbd1.out‚ 2.822265625 disk.dm-1.out‚ 205.8046875 disk.rbd2.in‚0* proc.memory.percent‚17.568230628967285 disk.drbd1777.in‚0 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚63833.71630859375 disk.rbd3.out‚0# proc.memory.rssMB‚ 2878.37890625 disk.rbd1.in‚0 disk.dm-0.in‚ 710.40625! disk.rbd2.out‚119.31298828125% disk./.usageGB‚1096.8843231201172 disk.sda3.out‚ 205.8046875 disk.dm-1.in‚0ZmZ2º- _wandbruntime‚5134  _runtime‚5134·âC{ɻà trainer/global_step‚6899 % _timestamp‚1.7884125242831776e+09 _runtime‚5134.695323721 _step‚563 ' lr-AdamW/pg1‚4.6881446605080575e-05 ' lr-AdamW/pg2‚4.6881446605080575e-05ZkؤÎ¼È ' lr-AdamW/pg1‚4.6881446605080575e-05 ' lr-AdamW/pg2‚4.6881446605080575e-05 trainer/global_step‚6899 % _timestamp‚1.7884125242831776e+09 _runtime‚5134.695323721 _step‚563³¡¬q@2½- _wandbruntime‚5134  _runtime‚5134!Å"F¾ØO @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚194352765420762 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 $ _timestamp‚1.788412524283884e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9696123094614 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9695957973210 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚194113813409792 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚184656824486912 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 230055715 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9724139508224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚215578080828928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205685678939136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚194113795406848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52957004 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 230075755 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚194352947581526 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚184656807447552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚203837765915136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52957006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 R>DeviceStatsMonitor.on_traN÷J˜ain_batch_start/active_bytes.all.freed‚203837765915136 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚194113813409792 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 139345608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _runtime‚5134.696339004 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205685654999040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9723970508288 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92061479 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚215578321645056 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90730145 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚194113795406848 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 139345608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚203837952918016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚6899 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚203837952918016 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9724139508224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39105018 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39104475 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9723970508288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985ž4_Iã'¿ÝO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9892425829888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚215578080828928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205685678939136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚194113795406848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52957004 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 230075755 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚194352947581526 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚184656807447552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚203837765915136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52957006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚203837765915136 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚194113813409792 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 139345608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _runtime‚5134.696339004 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205685654999040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9723970508288 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92061479 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚215578321645056 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90730145 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚194113795406848 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 139345608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚203837952918016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚6899 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚203837952918016 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9724139508224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39105018 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39104475 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9723970508288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚194352765420762 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 $ _timestamp‚1.788412524283884e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9696123094614 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9695957973210 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚194113813409792 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚184656824486912 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 230055715 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9724139508224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230108227 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚6899 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92074087 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90742806 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139345357 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52964383 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚194380279928614 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚194141082774528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9725363646464 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139365421 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚5135.352923406 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚215608406667264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚203866615434240 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9893838720000 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚203866428418048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52964381 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _step‚565 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚203866615434240 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚194141064771584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92074610 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205714567947264 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚184682767669248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 % _timestamp‚1.7884125249402492e+09 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9725363646464 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚215608651664384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 230108227 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9725532659712 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139365421 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39110227 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244997120 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚184682750629888 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 230088161 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚194380097767642 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944/1+¾GÂèM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚215608406667264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚203866615434240 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9893838720000 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚203866428418048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52964381 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _step‚565 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚203866615434240 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚194141064771584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92074610 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205714567947264 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚184682767669248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 % _timestamp‚1.7884125249402492e+09 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9725363646464 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚215608651664384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 230108227 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9725532659712 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139365421 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39110227 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244997120 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚184682750629888 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 230088161 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚194380097767642 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230108227 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚6899 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92074087 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90742806 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139345357 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52964383 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚194380279928614 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚194141082774528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9725363646464 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139365421 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚5135.352923406µv=2Ã- _wandbruntime‚5135  _runtime‚5135݉Ö[ÄÕ 0loss/FRt4/assignment_loss‚3.0943987369537354 .loss/FBt1/detection_loss‚2.448232378810644 /loss/FBt3/detection_loss‚1.0576947021412866 trainer/global_step‚6899 /loss/FRt3/assignment_loss‚2.393242420334565 2loss/SRqqt3/assignment_loss‚1.4174282289406455 1loss/FBt3/assignment_loss‚0.18083509909381842 2loss/SRqqt1/assignment_loss‚2.3120931647717953 0loss/FBt1/assignment_loss‚0.5102661127837109 /loss/FRt4/detection_loss‚2.1971718016660438 _step‚566 0loss/FRt2/assignment_loss‚3.0513869467771277 2loss/SRqqt4/assignment_loss‚1.7371147315277429 /loss/FBt4/detection_loss‚2.1516576260328293 /loss/FRt1/detection_loss‚3.2225036239624023 /loss/FBt2/detection_loss‚3.0671743197100505 epoch‚5 % _timestamp‚1.7884125249421458e+09 %loss/total_loss‚48.37260133491671 /loss/FRt1/assignment_loss‚5.176884298324585 0loss/FBt2/assignment_loss‚0.6694164025580643 .loss/FRt2/detection_loss‚2.124155188506504 .loss/FRt3/detection_loss‚2.294481297856883 1loss/SRqqt2/detection_loss‚1.4981990891520853 _runtime‚5135.354240487 2loss/SRqqt2/assignment_loss‚1.2888218824110635 0loss/FBt4/assignment_loss‚0.4070850330324447 1loss/SRqqt3/detection_loss‚1.5897842967374758 1loss/SRqqt4/detection_loss‚1.7521748210563035 1loss/SRqqt1/detection_loss‚2.7303991317749023b•†`ÅÚ 0loss/FRt4/assignment_loss‚3.0943987369537354 .loss/FBt1/detection_loss‚2.448232378810644 /loss/FBt3/detection_loss‚1.0576947021412866 trainer/global_step‚6899 /loss/FRt3/assignment_loss‚2.393242420334565 2loss/SRqqt3/assignment_loss‚1.4174282289406455 1loss/FBt3/assignment_loss‚0.18083509909381842 2loss/SRqqt1/assignment_loss‚2.3120931647717953 0loss/FBt1/assignment_loss‚0.5102661127837109 /loss/FRt4/detection_loss‚2.1971718016660438 _step‚566 0loss/FRt2/assignment_loss‚3.0513869467771277 2loss/SRqqt4/assignment_loss‚1.7371147315277429 /loss/FBt4/detection_loss‚2.1516576260328293 /loss/FRt1/detection_loss‚3.2225036239624023 /loss/FBt2/detection_loss‚3.0671743197100505 epoch‚5 % _timestamp‚1.7884125249421458e+09 %loss/total_loss‚48.37260133491671 /loss/FRt1/assignment_loss‚5.176884298324585 0loss/FBt2/assignment_loss‚0.6694164025580643 .loss/FRt2/detection_loss‚2.124155188506504 .loss/FRt3/detection_loss‚2.294481297856883 1loss/SRqqt2/detection_loss‚1.4981990891520853 _runtime‚5135.354240487 2loss/SRqqt2/assignment_loss‚1.2888218824110635 0loss/FBt4/assignment_loss‚0.4070850330324447 1loss/SRqqt3/detection_loss‚1.5897842967374758 1loss/SRqqt4/detection_loss‚1.7521748210563035 1loss/SRqqt1/detection_loss‚2.7303991317749023¶8jÞÆ‚(:„ ö„äÔàüË™ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.48' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.896 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾Â·%Ç‚(:š ö„äÔúÓñ›# disk.dm-0.out‚63834.07568359375# memory_percent‚98.4778642654419 cpu‚25.35768897194014% disk./.usageGB‚1097.0098304748535 disk.sda3.out‚ 206.2890625 disk.sda.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 206.2890625 disk.rbd3.in‚0 network.recv‚ 1.905447e+06! disk.rbd2.out‚119.31298828125 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.in‚0) disk./.usagePercent‚73.62125017572357 disk.sda1.out‚0 disk.sda2.out‚0 network.sent‚ 1.59708e+07 disk.drbd1777.in‚0 disk.rbd1.out‚ 2.822265625 disk.rbd0.out‚0# proc.memory.rssMB‚ 2878.37890625 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 710.40625 disk.rbd3.out‚0( disk.nvme0c0n1.out‚63834.07568359375* proc.memory.percent‚17.568230628967285 disk.sda1.in‚0& disk.nvme0n1.out‚63834.07568359375 disk.rbd0.in‚0 disk.rbd2.in‚0( proc.memory.availableMB‚ 249.38671875 proc.cpu.threads‚17 disk.sda.out‚ 206.2890625 disk.dm-0.in‚ 710.40625 disk.dm-1.in‚0+ùpÈ‚(:… ……äÔØòŒ™ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.353' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0706 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚŽÉ‚(:” ……äÔ¶…Äœ proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚63933.7470703125 disk.sda2.out‚0 disk.sda.out‚ 206.84375 disk.nvme0n1.in‚ 710.40625 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.in‚0 cpu‚25.356792327241312 network.recv‚ 1.911864e+06( disk./.usagePercent‚73.6212145906763 disk.rbd3.in‚0 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2878.37890625* proc.memory.percent‚17.568230628967285 network.sent‚ 1.6079433e+07 disk.dm-1.out‚ 206.84375 disk.rbd3.out‚0( proc.memory.availableMB‚ 249.39453125! disk.rbd2.out‚119.45361328125 disk.nvme0c0n1.in‚ 710.40625 disk.sda3.out‚ 206.84375 disk.rbd1.out‚ 2.822265625 disk.dm-0.in‚ 710.40625 disk.drbd1777.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚63933.7470703125" disk.dm-0.out‚63933.7470703125% disk./.usageGB‚1097.0093002319336 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚98.47781658172607Û}r2Ê- _wandbruntime‚5169  _runtime‚5169þ/Ñ¡ÈË _step‚567 ' lr-AdamW/pg1‚4.6154466057402115e-05 ' lr-AdamW/pg2‚4.6154466057402115e-05 trainer/global_step‚6949 $ _timestamp‚1.788412558913939e+09 _runtime‚5169.326234559~xYÌÍÌÇ _runtime‚5169.326234559 _step‚567 ' lr-AdamW/pg1‚4.6154466057402115e-05 ' lr-AdamW/pg2‚4.6154466057402115e-05 trainer/global_step‚6949 $ _timestamp‚1.788412558913939e+09· ¼2Í- _wandbruntime‚5169  _runtime‚5169».zß'ÎÙO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9793641830912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92703970 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9963530278400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140318272 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚205270905474560 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚195477281646592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231681367 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9765594913022 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92703424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91363097 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚217094617909760 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚195719396355970 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚185953966564352 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91363097 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9793810830848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53325854 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39378114 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9765429791618 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 trainer/global_step‚6949 % _timestamp‚1.7884125589147422e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231701407 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91363095 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140338310 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚195719578516734 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚205271092477440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9963315499520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚185953983603712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9793810830848 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9793641830912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚207131302410240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140338310 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚207131326350336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚207131302410240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140338310 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚207131326350336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚205270905474560 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚195477281646592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231681367 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9765594913022 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92703424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91363097 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚217094617909760 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚195719396355970 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚185953966564352 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91363097 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9793810830848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53325854 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39378114 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9765429791618 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 trainer/global_step‚6949 % _timestamp‚1.7884125589147422e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231701407 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91363095 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140338310 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚195719578516734 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚205271092477440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9963315499520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚185953983603712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0¸ß”ëã2Ð- _wandbruntime‚5169  _runtime‚5169¾‘JÜ«ÑãM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140358123 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚195504551011328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚217124942051328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91375754 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚195746728702850 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 trainer/global_step‚6949 _step‚569 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140338059 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚5169.980790635 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231733879 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92716613 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚,Û*> 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92716088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140338059 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚195504551011328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9964726692864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231733879 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9766818956162 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9795203982336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9795034969088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91375754 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53333231 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39383380 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9795034969088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9766984077774 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91375756 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚205299567977472 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚207160239298560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92716088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140338059 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚195504551011328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9964726692864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231733879 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9766818956162 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9795203982336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9795034969088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91375754 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53333231 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39383380 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9795034969088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9766984077774 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91375756 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚205299567977472 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚207160239298560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140358123 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚195504551011328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚217124942051328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91375754 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚195746728702850 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 trainer/global_step‚6949 _step‚569 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140338059 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚5169.980790635 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231733879 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92716613 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408¹«ãr©2Ó- _wandbruntime‚5169  _runtime‚5169DWÐWÔÑ 2loss/SRqqt4/assignment_loss‚2.0103109346135803 /loss/FRt3/detection_loss‚1.9879891463416688 0loss/FRt1/assignment_loss‚4.6352687192999795 /loss/FRt2/assignment_loss‚3.308258401023017 1loss/SRqqt2/assignment_loss‚1.487306026374235 0loss/FBt2/assignment_loss‚0.8596746549301315 .loss/FRt1/detection_loss‚3.168951734252598 /loss/FBt3/detection_loss‚0.7362782720737605 1loss/SRqqt1/assignment_loss‚3.003790406080393 2loss/SRqqt3/assignment_loss‚1.3028587348245342 0loss/FBt1/assignment_loss‚0.8059340998968563 0loss/FBt3/assignment_loss‚0.2583918237149335 /loss/FRt2/detection_loss‚2.2000853856404623 0loss/SRqqt2/detection_loss‚1.385712974655785 1loss/SRqqt4/detection_loss‚1.7033870067933332 /loss/FBt4/detection_loss‚2.1505059979178687 /loss/FRt4/detection_loss‚2.2423936245488187 1loss/SRqqt3/detection_loss‚1.1926830932497978 epoch‚5 % _timestamp‚1.7884125595699298e+09 _runtime‚5169.982018077 0loss/FRt3/assignment_loss‚2.1763835473263518 0loss/FBt4/assignment_loss‚0.6307725692597821 trainer/global_step‚6949 0loss/FRt4/assignment_loss‚3.2680982164308134 /loss/SRqqt1/detection_loss‚2.47681311919139 .loss/FBt1/detection_loss‚2.995950400829315 /loss/FBt2/detection_loss‚3.3843006253242494 %loss/total_loss‚49.37209951459366 _step‚570Â>Ü\ÕÖ % _timestamp‚1.7884125595699298e+09 _runtime‚5169.982018077 0loss/FRt3/assignment_loss‚2.1763835473263518 0loss/FBt4/assignment_loss‚0.6307725692597821 trainer/global_step‚6949 0loss/FRt4/assignment_loss‚3.2680982164308134 /loss/SRqqt1/detection_loss‚2.47681311919139 .loss/FBt1/detection_loss‚2.995950400829315 /loss/FBt2/detection_loss‚3.3843006253242494 %loss/total_loss‚49.37209951459366 _step‚570 2loss/SRqqt4/assignment_loss‚2.0103109346135803 /loss/FRt3/detection_loss‚1.9879891463416688 0loss/FRt1/assignment_loss‚4.6352687192999795 /loss/FRt2/assignment_loss‚3.308258401023017 1loss/SRqqt2/assignment_loss‚1.487306026374235 0loss/FBt2/assignment_loss‚0.8596746549301315 .loss/FRt1/detection_loss‚3.168951734252598 /loss/FBt3/detection_loss‚0.7362782720737605 1loss/SRqqt1/assignment_loss‚3.003790406080393 2loss/SRqqt3/assignment_loss‚1.3028587348245342 0loss/FBt1/assignment_loss‚0.8059340998968563 0loss/FBt3/assignment_loss‚0.2583918237149335 /loss/FRt2/detection_loss‚2.2000853856404623 0loss/SRqqt2/detection_loss‚1.385712974655785 1loss/SRqqt4/detection_loss‚1.7033870067933332 /loss/FBt4/detection_loss‚2.1505059979178687 /loss/FRt4/detection_loss‚2.2423936245488187 1loss/SRqqt3/detection_loss‚1.1926830932497978 epoch‚5ºÎj–Ö‚(:„ ”…äԌřš gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚91.365' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.273 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06¦U#ׂ(:˜ ”…äÔУíœ# proc.memory.rssMB‚ 2878.37890625( disk./.usagePercent‚73.6296034455583 disk.rbd2.in‚0! disk.rbd2.out‚119.45361328125 disk.rbd3.out‚0 disk.drbd1777.in‚0% disk.nvme0n1.out‚64032.8330078125 network.sent‚ 1.608479e+07 network.recv‚ 1.915742e+06% disk./.usageGB‚1097.1343002319336 disk.sda3.out‚ 207.3671875 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda3.in‚0 disk.drbd1777.out‚0" disk.dm-0.out‚64032.8330078125 disk.sda.in‚0 disk.nvme0n1.in‚ 710.40625 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚98.47512245178223 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.out‚ 2.822265625 disk.rbd0.in‚0* proc.memory.percent‚17.568230628967285 disk.dm-0.in‚ 710.40625 disk.dm-1.out‚ 207.3671875 disk.sda2.in‚0 disk.sda.out‚ 207.3671875' disk.nvme0c0n1.out‚64032.8330078125 disk.sda1.in‚0' proc.memory.availableMB‚ 249.8359375 cpu‚25.355904073934084sírØ‚(:„ £…äÔÜÜ°™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.48' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.896 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èÄ Ô&Ù‚(:› £…äÔ«¤ãœ disk.sda.out‚ 208.11328125) disk./.usagePercent‚73.64638115532227 disk.rbd1.in‚0 cpu‚25.355024862957354 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 208.11328125$ memory_percent‚98.47447872161865 network.sent‚ 1.619265e+07 network.recv‚ 1.921338e+06 disk.rbd1.out‚ 2.822265625 disk.dm-0.in‚ 710.40625 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚64132.0517578125* proc.memory.percent‚17.568254470825195 proc.cpu.threads‚17% disk./.usageGB‚1097.3843002319336 disk.drbd1777.out‚0 disk.dm-1.out‚ 208.11328125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.out‚0( proc.memory.availableMB‚ 249.94140625" proc.memory.rssMB‚ 2878.3828125 disk.drbd1777.in‚0' disk.nvme0c0n1.out‚64132.0517578125 disk.sda2.in‚0" disk.dm-0.out‚64132.0517578125 disk.nvme0n1.in‚ 710.40625 disk.rbd2.out‚119.6181640625ïjtÔ2Ú- _wandbruntime‚5204  _runtime‚5204\R8AÇÛÁ % _timestamp‚1.7884125936121047e+09 _runtime‚5204.024281322 _step‚571 & lr-AdamW/pg1‚4.542839473602481e-05 & lr-AdamW/pg2‚4.542839473602481e-05 trainer/global_step‚6999e“¥ÌÜÆ & lr-AdamW/pg2‚4.542839473602481e-05 trainer/global_step‚6999 % _timestamp‚1.7884125936121047e+09 _runtime‚5204.024281322 _step‚571 & lr-AdamW/pg1‚4.542839473602481e-05»«Î2Ý- _wandbruntime‚5204  _runtime‚5204éSšá'ÞÛO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53694706 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233327059 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39649467 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93344711 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚196840749883392 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9834901610026 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚187251142720512 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9835066731430 trainer/global_step‚6999 _step‚572 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9863313153536 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10034420488704 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚187251125681152 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39650005 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚218611159725568 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚208576949821440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚197086209451942 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141331012 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141310974 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚206704045033984 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚206704045033984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9863482153472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53694704 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚5204.025290002 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9863482153472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141310974 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91996045 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚218611394250240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚196840749883392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚206704232036864 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚196840731880448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884125936128743e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9863313153536 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚208576973761536 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0ü-=GßàO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚206704232036864 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.laÌíæyŸ&rge_pool.freed‚196840731880448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884125936128743e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9863313153536 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚208576973761536 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53694706 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233327059 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39649467 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93344711 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚196840749883392 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9834901610026 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚187251142720512 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9835066731430 trainer/global_step‚6999 _step‚572 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9863313153536 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10034420488704 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚187251125681152 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39650005 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚218611159725568 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚208576949821440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚197086209451942 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141331012 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141310974 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚206704045033984 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚206704045033984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9863482153472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53694704 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚5204.025290002 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9863482153472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141310974 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91996045 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚218611394250240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚196840749883392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0¼P=ãÂ2à- _wandbruntime‚5204  _runtime‚5204nQ˜3ë&áåM H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233359531 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9836455896182 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 % _timestamp‚1.7884125942590723e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233339465 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93356979 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53702083 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚206732707536896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92008706 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚187277068863488 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141330761 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9864875304960 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233359531 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚206732894553088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚196868001245184 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92008706 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53702081 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚534 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚208605886709760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚206732894553088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9864706291712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141350825 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚196868019248128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9864875304960 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9864706291712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚187277085902848 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39654898 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10035835998208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 trainer/global_step‚6999 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141350825 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚196868019248128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39655430 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚218641490293760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚208605862769664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚573 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚197113359638058 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92008704 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92008704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0ß\Xð&âêM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141350825 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚196868019248128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39655430 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚218641490293760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚208605862769664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚573 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚197113359638058 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92008704 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92008704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233359531 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9836455896182 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 % _timestamp‚1.7884125942590723e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233339465 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93356979 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53702083 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚206732707536896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92008706 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚187277068863488 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141330761 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9864875304960 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233359531 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚206732894553088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚196868001245184 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92008706 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53702081 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚534 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚208605886709760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚206732894553088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9864706291712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141350825 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚196868019248128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9864875304960 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9864706291712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚187277085902848 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39654898 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10035835998208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 trainer/global_step‚6999½$Jzˆ2ã- _wandbruntime‚5204  _runtime‚5204'ÈÛXäÒ /loss/FRt1/detection_loss‚3.1366096351827895 /loss/FRt3/detection_loss‚2.1577988779399453 _step‚574 0loss/FRt3/assignment_loss‚1.8918001390084989 0loss/SRqqt3/assignment_loss‚1.66010386745135 0loss/FBt2/assignment_loss‚1.1269290844599407 /loss/FRt4/detection_loss‚2.0752775845478992 .loss/FBt1/detection_loss‚3.317667746543884 0loss/FRt2/assignment_loss‚3.0410501077061607 /loss/FRt4/assignment_loss‚3.524592865486534 1loss/SRqqt2/assignment_loss‚1.430618007612579 1loss/SRqqt2/detection_loss‚1.5258705819792606 0loss/SRqqt4/detection_loss‚1.735872795707301 epoch‚5 /loss/FRt1/assignment_loss‚4.926781424454281 1loss/FBt1/assignment_loss‚0.29654638858137333 _runtime‚5204.673078304 2loss/SRqqt1/assignment_loss‚2.5342781058064214 1loss/SRqqt3/detection_loss‚1.3556490307837201 &loss/total_loss‚49.883675360343034 trainer/global_step‚6999 2loss/SRqqt4/assignment_loss‚1.9732234650536586 .loss/FBt2/detection_loss‚2.669673611720403 .loss/FBt4/detection_loss‚2.745170161128044 % _timestamp‚1.7884125942606878e+09 0loss/FBt4/assignment_loss‚0.6981124021021969 /loss/FBt3/detection_loss‚0.7903930136793587 1loss/FBt3/assignment_loss‚0.18884652897804077 /loss/FRt2/detection_loss‚2.3726896899087087 0loss/SRqqt1/detection_loss‚2.708120244520682"_œ]å× 2loss/SRqqt1/assignment_loss‚2.5342781058064214 1loss/SRqqt3/detection_loss‚1.3556490307837201 &loss/total_loss‚49.883675360343034 trainer/global_step‚6999 2loss/SRqqt4/assignment_loss‚1.9732234650536586 .loss/FBt2/detection_loss‚2.669673611720403 .loss/FBt4/detection_loss‚2.745170161128044 % _timestamp‚1.7884125942606878e+09 0loss/FBt4/assignment_loss‚0.6981124021021969 /loss/FBt3/detection_loss‚0.7903930136793587 1loss/FBt3/assignment_loss‚0.18884652897804077 /loss/FRt2/detection_loss‚2.3726896899087087 0loss/SRqqt1/detection_loss‚2.708120244520682 /loss/FRt1/detection_loss‚3.1366096351827895 /loss/FRt3/detection_loss‚2.1577988779399453 _step‚574 0loss/FRt3/assignment_loss‚1.8918001390084989 0loss/SRqqt3/assignment_loss‚1.66010386745135 0loss/FBt2/assignment_loss‚1.1269290844599407 /loss/FRt4/detection_loss‚2.0752775845478992 .loss/FBt1/detection_loss‚3.317667746543884 0loss/FRt2/assignment_loss‚3.0410501077061607 /loss/FRt4/assignment_loss‚3.524592865486534 1loss/SRqqt2/assignment_loss‚1.430618007612579 1loss/SRqqt2/detection_loss‚1.5258705819792606 0loss/SRqqt4/detection_loss‚1.735872795707301 epoch‚5 /loss/FRt1/assignment_loss‚4.926781424454281 1loss/FBt1/assignment_loss‚0.29654638858137333 _runtime‚5204.673078304¾IXó†5æ‚(:† ²…äÔ—ÕØš gpu.0.gpu‚23.0 gpu.¬ný\0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚106.881' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3762 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿Še–ç‚(:” ²…äÔÝ컜 disk.sda3.out‚ 208.609375 disk.rbd3.out‚0' disk.nvme0c0n1.out‚64132.3408203125 disk.sda.out‚ 208.609375 disk.rbd3.in‚0 disk.sda3.in‚0 cpu‚25.354152948522316 network.sent‚ 1.6198888e+07 disk.drbd1777.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 208.609375 disk.sda2.out‚0$ memory_percent‚98.47707748413086* proc.memory.percent‚17.568254470825195 disk.rbd1.in‚0 disk.dm-0.in‚ 710.40625 disk.drbd1777.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚121.2001953125 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.in‚0% disk.nvme0n1.out‚64132.3408203125 network.recv‚ 1.926163e+06) disk./.usagePercent‚73.64638115532227 disk.sda.in‚0 disk.rbd1.out‚ 2.822265625 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd2.in‚0& proc.memory.availableMB‚ 249.515625" proc.memory.rssMB‚ 2878.3828125% disk./.usageGB‚1097.3843002319336 disk.nvme0n1.in‚ 710.40625" disk.dm-0.out‚64132.3408203125 proc.cpu.threads‚17 disk.sda1.in‚0°¯œè‚(:‘ Á…äÔ¹¥‰™ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.611' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.322200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûÓç¸'é‚(:œ Á…äԟDz› disk.rbd3.out‚0 disk.sda3.out‚ 209.11328125" proc.memory.rssMB‚ 2878.3828125) disk./.usagePercent‚73.63870323336286 disk.dm-1.out‚ 209.11328125' proc.memory.availableMB‚ 249.3515625* proc.memory.percent‚17.568254470825195 network.recv‚ 1.932514e+06 disk.sda.out‚ 209.11328125& disk.nvme0n1.out‚64231.69091796875 disk.rbd0.in‚0 disk.sda2.in‚0 cpu‚25.3532462797604 disk.sda.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.out‚0 disk.sda3.in‚0 network.sent‚ 1.6307658e+07 disk.drbd1777.out‚0 disk.dm-0.in‚ 710.40625 disk.rbd2.out‚121.2783203125$ memory_percent‚98.47807884216309% disk./.usageGB‚1097.2698936462402 disk.rbd2.in‚0( disk.nvme0c0n1.out‚64231.69091796875 disk.rbd1.in‚0 disk.nvme0n1.in‚ 710.40625 disk.rbd0.out‚0 proc.cpu.threads‚17# disk.dm-0.out‚64231.69091796875 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.rbd1.out‚ 2.822265625½u¥ ê‚(:… Ð…äÔÐÛÊ™ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚94.041' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8082 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZF ªë‚(:‹ Ð…äÔ‰Éù›) disk./.usagePercent‚73.65549220745835# disk./.usageGB‚1097.52006149292 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd2.out‚121.2783203125* proc.memory.percent‚17.568254470825195 disk.rbd1.out‚ 2.822265625 disk.rbd0.out‚0$ memory_percent‚98.47660064697266 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2878.3828125 cpu‚25.352375797168907 disk.sda.out‚ 209.609375 disk.rbd1.in‚0 disk.dm-0.out‚64331.03515625 disk.rbd3.in‚0% disk.nvme0c0n1.out‚64331.03515625 disk.sda3.in‚0 disk.sda3.out‚ 209.609375% proc.memory.availableMB‚ 249.59375 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 710.40625 disk.dm-0.in‚ 710.40625 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 1.6312971e+07 disk.drbd1777.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 209.609375 disk.nvme0c0n1.in‚ 710.40625# disk.nvme0n1.out‚64331.03515625 network.recv‚ 1.936375e+06!FD¹2ì- _wandbruntime‚5238  _runtime‚5238·]ìAÉíà _step‚575 ' lr-AdamW/pg1‚4.4703383953123746e-05 ' lr-AdamW/pg2‚4.4703383953123746e-05 trainer/global_step‚7049 % _timestamp‚1.7884126279900806e+09 _runtime‚5238.402246085pn)ÎîÈ ' lr-AdamW/pg2‚4.4703383953123746e-05 trainer/global_step‚7049 % _timestamp‚1.7884126279900806e+09 _runtime‚5238.402246085 _step‚575 ' lr-AdamW/pg1‚4.4703383953123746e-05¿U1Ýó2ï- _wandbruntime‚5238  _runtime‚5238zh¶üà'ðÚO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9933153476096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54063554 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39921650 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚198452658226386 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚5238.403232523 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142303676 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54063556 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚220127756401664 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚198204218120192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9904538549838 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93985206 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚546 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93984658 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10105369753600 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142303676 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚548 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚220127990926336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _step‚576 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92628995 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚208137184593408 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142323714 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9904373428434 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92628997 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚208137371596288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198204200117248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚208137184593408 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚210022621172736 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92628997 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39921104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚210022597232640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚208137371596288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788412627990842e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚7049 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚198204200117248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198204218120192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9932984476160 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0z…1å'ñßO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9932984476160 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10105159169024 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚198452840387150 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚198452658226386 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚5238.403232523 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142303676 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54063556 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚220127756401664 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚198204218120192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9904538549838 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93985206 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚546 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93984658 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10105369753600 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142303676 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚548 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚220127990926336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _step‚576 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92628995 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚208137184593408 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142323714 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9904373428434 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92628997 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚208137371596288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198204200117248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚208137184593408 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚210022621172736 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92628997 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39921104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚210022597232640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚208137371596288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788412627990842e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚7049 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚198204200117248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198204218120192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9932984476160 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9933153476096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54063554 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39921650 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142343527 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚210051510180864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93997979 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚198479990573266 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 234985183 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9905927714590 trainer/global_step‚7049 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142343527 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 234985183 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚188574227980288 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92641656 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚208166034112512 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚198480172734238 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9934377614336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9934377614336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92641654 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54070931 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚208166034112512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚208165847096320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10106787088896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 234965117 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚188574245019648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93997443 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39927046 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7884126286314516e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚188574245019648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93997443 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39927046 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7884126286314516e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142343527 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚210051510180864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93997979 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚198479990573266 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 234985183 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9905927714590 trainer/global_step‚7049 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142343527 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 234985183 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚188574227980288 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92641656 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚208166034112512 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚198480172734238 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9934377614336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9934377614336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92641654 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54070931 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚208166034112512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚208165847096320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224ÁÓ$2õ- _wandbruntime‚5239  _runtime‚5239Ff™ðVöÐ _step‚578 1loss/SRqqt1/assignment_loss‚2.667859259404634 /loss/FBt2/assignment_loss‚0.352626964993154 1loss/SRqqt1/detection_loss‚3.4157781381356087 2loss/SRqqt4/assignment_loss‚1.2901409772009804 epoch‚5 trainer/global_step‚7049 0loss/FRt2/assignment_loss‚2.9042851821235987 0loss/FBt1/assignment_loss‚0.6379561270863027 1loss/FBt3/assignment_loss‚0.34041372089224176 0loss/SRqqt4/detection_loss‚1.732434495137288 /loss/FRt2/detection_loss‚2.3081343990305196 /loss/FRt4/detection_loss‚2.1160464232618157 1loss/SRqqt2/detection_loss‚1.5793297160732545 0loss/SRqqt3/detection_loss‚1.267571722125185 2loss/SRqqt3/assignment_loss‚1.4200578647954711 /loss/FBt4/assignment_loss‚0.481222636981069 /loss/FRt1/detection_loss‚3.1492171597480776 /loss/FBt4/detection_loss‚2.2096893319061826 % _timestamp‚1.7884126286330285e+09 _runtime‚5239.045354548 /loss/FRt4/assignment_loss‚3.267072069644928 2loss/SRqqt2/assignment_loss‚1.6740987377272825 /loss/FRt3/detection_loss‚1.9014020521266788 .loss/FBt1/detection_loss‚2.703562473257383 /loss/FBt3/detection_loss‚0.6937387864494517 %loss/total_loss‚49.69711288767067 /loss/FRt1/assignment_loss‚5.371015105247498 /loss/FRt3/assignment_loss‚2.111955138982511 .loss/FBt2/detection_loss‚4.101504405339559Q%[[÷Õ /loss/FRt1/detection_loss‚3.1492171597480776 /loss/FBt4/detection_loss‚2.2096893319061826 % _timestamp‚1.7884126286330285e+09 _runtime‚5239.045354548 /loss/FRt4/assignment_loss‚3.267072069644928 2loss/SRqqt2/assignment_loss‚1.6740987377272825 /loss/FRt3/detection_loss‚1.9014020521266788 .loss/FBt1/detection_loss‚2.703562473257383 /loss/FBt3/detection_loss‚0.6937387864494517 %loss/total_loss‚49.69711288767067 /loss/FRt1/assignment_loss‚5.371015105247498 /loss/FRt3/assignment_loss‚2.111955138982511 .loss/FBt2/detection_loss‚4.101504405339559 _step‚578 1loss/SRqqt1/assignment_loss‚2.667859259404634 /loss/FBt2/assignment_loss‚0.352626964993154 1loss/SRqqt1/detection_loss‚3.4157781381356087 2loss/SRqqt4/assignment_loss‚1.2901409772009804 epoch‚5 trainer/global_step‚7049 0loss/FRt2/assignment_loss‚2.9042851821235987 0loss/FBt1/assignment_loss‚0.6379561270863027 1loss/FBt3/assignment_loss‚0.34041372089224176 0loss/SRqqt4/detection_loss‚1.732434495137288 /loss/FRt2/detection_loss‚2.3081343990305196 /loss/FRt4/detection_loss‚2.1160464232618157 1loss/SRqqt2/detection_loss‚1.5793297160732545 0loss/SRqqt3/detection_loss‚1.267571722125185 2loss/SRqqt3/assignment_loss‚1.4200578647954711 /loss/FBt4/assignment_loss‚0.481222636981069Â9`Œø‚(: ß…äÔõ鸙 gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.0' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àö1þ%ù‚(:š ß…äÔ”˜Žœ disk.rbd0.in‚0! disk.rbd2.out‚121.45458984375 disk.dm-0.in‚ 710.40625$ memory_percent‚98.47702980041504 cpu‚25.351466903955856) disk./.usagePercent‚73.65554878512341 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 710.40625 disk.sda.out‚ 210.37109375 proc.cpu.threads‚17 disk.rbd3.in‚0& disk.nvme0c0n1.out‚64430.611328125 disk.drbd1777.in‚0! disk.dm-0.out‚64430.611328125 disk.rbd1.out‚ 2.822265625% disk./.usageGB‚1097.5209045410156 disk.dm-1.out‚ 210.37109375 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 249.5234375 network.recv‚ 1.939303e+06 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2878.38671875 network.sent‚ 1.6317495e+07$ disk.nvme0n1.out‚64430.611328125 disk.rbd0.out‚0 disk.sda3.out‚ 210.37109375 disk.sda.in‚0 disk.nvme0n1.in‚ 710.40625* proc.memory.percent‚17.568278312683105 disk.dm-1.in‚0¤Å¦ì‘ú‚(:† î…äÔòŒ›™ gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.584' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5168 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í’·Sû‚(:‰ î…äÔáôÖ›! disk.dm-0.out‚64430.857421875 disk.dm-1.out‚210.875 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.out‚0$ memory_percent‚98.47612380981445# proc.memory.rssMB‚ 2878.38671875 disk.rbd3.out‚0& disk.nvme0c0n1.out‚64430.857421875 disk.sda2.in‚0) disk./.usagePercent‚73.68923963263697% disk./.usageGB‚1098.0229225158691 disk.sda.out‚210.875 disk.dm-1.in‚0& proc.memory.availableMB‚ 249.671875 disk.drbd1777.out‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 2.822265625* proc.memory.percent‚17.568278312683105 network.sent‚ 1.6426247e+07 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda3.out‚210.875 disk.sda.in‚0$ disk.nvme0n1.out‚64430.857421875! disk.rbd2.out‚121.45458984375 network.recv‚ 1.946216e+06 disk.sda1.out‚0 disk.dm-0.in‚ 710.40625 disk.rbd0.out‚0 cpu‚25.35066358598117 proc.cpu.threads‚17 disk.nvme0n1.in‚ 710.40625Îò2ü- _wandbruntime‚5273  _runtime‚5273ù%k:ÇýÁ & lr-AdamW/pg1‚4.397958479985944e-05 & lr-AdamW/pg2‚4.397958479985944e-05 trainer/global_step‚7099 % _timestamp‚1.7884126627369368e+09 _runtime‚5273.149186517 _step‚579cn¶œÌþÆ % _timestamp‚1.7884126627369368e+09 _runtime‚5273.149186517 _step‚579 & lr-AdamW/pg1‚4.397958479985944e-05 & lr-AdamW/pg2‚4.397958479985944e-05 trainer/global_step‚7099ÃgnW¸2ÿ- _wandbruntime‚5273  _runtime‚5273€%øå'€ßO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40191478 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54432406 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚209570511155712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94623339 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚5273.150093055 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚211468268583936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143296378 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94623884 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93261947 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40190935 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10002655798784 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚209570511155712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211468244643840 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚221644583640064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10002655798784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚209570324152832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9974010368246 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 236558323 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚199567668354048 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚189845460954112 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93261947 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 236578363 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10002824798720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93261945 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143316416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚209570324152832 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚189845460954112 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93261947 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 236578363 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10002824798720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93261945 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143316416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚209570324152832 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93261947 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40190935 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10002655798784 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚209570511155712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211468244643840 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚221644583640064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10002655798784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚209570324152832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9974010368246 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 236558323 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚199567668354048 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10004048936960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚199594937718784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93274606 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10004217950208 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94636045 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 236610835 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40196264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚199594955721728 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚209599173671936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10004048936960 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9975234411386 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10004217950208 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211497157592064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10177521419264 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚199594937718784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 236610835 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93274604 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚209599173671936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143336229 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚527 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143316165 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40196791 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10177734087680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _step‚581 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 236590769 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 143336229 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚199594955721728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54439783 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚02<µô&„îM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚209598986655744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚529 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9975234411386 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10004217950208 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211497157592064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10177521419264 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚199594937718784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 236610835 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93274604 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚209599173671936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143336229 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚527 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143316165 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40196791 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10177734087680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _step‚581 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 236590769 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 143336229 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚199594955721728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54439783 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 236590769 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚211497181532160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚189871387097088 trainer/global_step‚7099 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7884126633873343e+09 _runtime‚5273.800182398 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10004048936960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚199594937718784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93274606 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10004217950208 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94636045 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 236610835 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40196264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚199594955721728 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚209599173671936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10004048936960 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10072327121408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93894895 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93894897 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚555 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10072327121408 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95263951 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚191142603031552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚553 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40462142 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144309118 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 _step‚584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚200931154593792 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚211003650715136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚211003463712256 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚211003650715136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93894897 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238204015 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10043482186654 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚223161127906816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54801256 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚201186102257566 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95263396 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10247211911680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144289080 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚211003463712256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚200931154593792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10072496121344 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10247007618560 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚5307.895304094 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚212913915995136Ÿßþì'‘æO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚553 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40462142 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144309118 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 _step‚584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚200931154593792 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚211003650715136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚211003463712256 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚211003650715136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93894897 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238204015 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10043482186654 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚223161127906816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54801256 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚201186102257566 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95263396 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10247211911680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144289080 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚211003463712256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚200931154593792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10072496121344 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10247007618560 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚5307.895304094 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚212913915995136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚212913892055040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10072327121408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93894895 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93894897 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚555 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10072327121408 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95263951 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚191142603031552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238216421 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10073889272832 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚200958423958528 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95276083 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144328931 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144308867 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40467452 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚201213434604654 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚200958423958528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10073720259584 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93907556 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144328931 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238236487 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93907556 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144308867 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚223191462258688 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚223191234038784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238236487 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238216421 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93907554 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54808631 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚212942805003264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10073720259584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54808633 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10248429035520 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚191168546213888 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238216421 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10073889272832 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚200958423958528 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95276083 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144328931 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144308867 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40467452 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚201213434604654 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚200958423958528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10073720259584 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93907556 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144328931 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238236487 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93907556 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144308867 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚223191462258688 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚223191234038784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238236487 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238216421 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93907554 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54808631 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚212942805003264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10073720259584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54808633 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10248429035520 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚191168546213888 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚212436603271680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95903059 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55170104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10112954005062 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 239809627 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94527847 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚192439779187712 % _timestamp‚1.7884127319684074e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 239829667 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚224677477358080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚202294622830592 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94527847 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚192439762148352 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚212436790274560 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880B±.)V£æO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55170104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10112954005062 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 239809627 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94527847 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚192439779187712 % _timestamp‚1.7884127319684074e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 239829667 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚224677477358080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚202294622830592 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94527847 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚192439762148352 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚212436790274560 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚202294604827648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145301820 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚212436603271680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95903059ÌLà’¿2¤- _wandbruntime‚5343  _runtime‚5343•4›íñ&¥ëM OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145301569 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚214388476354560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10143560595456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40739083 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚505 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10319354107904 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚202580065539862 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚212465265774592 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚214388452414464 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10114178048202 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚202321874192384 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145321633 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚202321892195328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚212465452790784 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145321633 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94540504 _runtime‚5343.025010583 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10143560595456 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 239842073 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10143391582208 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _step‚589 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10319568873472 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 239842073 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40738578 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚192465705330688 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55177483 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 239862139 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚224707806522368 % _timestamp‚1.7884127326124268e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 239862139 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚192465722370048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚202321892195328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10114343169814 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94540506 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95916059 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚224708045228032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚202579883378890 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55177481 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95916566 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚507 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94540504 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0.gšö&¦ðM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94540504 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145301569 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚214388476354560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10143560595456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40739083 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚505 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10319354107904 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚202580065539862 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚212465265774592 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚214388452414464 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10114178048202 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚202321874192384 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145321633 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚202321892195328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚212465452790784 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145321633 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94540504 _runtime‚5343.025010583 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10143560595456 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 239842073 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10143391582208 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _step‚589 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10319568873472 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 239842073 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40738578 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚192465705330688 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55177483 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 239862139 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚224707806522368 % _timestamp‚1.7884127326124268e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 239862139 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚192465722370048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚202321892195328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10114343169814 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94540506 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95916059 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚224708045228032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚202579883378890 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55177481 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95916566 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚507 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7199Í8— õ2§- _wandbruntime‚5343  _runtime‚5343ofÄZ¨Ô $ _timestamp‚1.788412732613993e+09 %loss/total_loss‚47.65461220993972 /loss/FRt1/assignment_loss‚4.871151685714722 /loss/FRt2/assignment_loss‚2.510581812437843 0loss/SRqqt1/detection_loss‚2.399276981548387 /loss/FBt3/detection_loss‚0.9891267180273438 2loss/SRqqt4/assignment_loss‚1.5911982493406092 0loss/FBt3/assignment_loss‚0.1890070190831025 1loss/SRqqt2/detection_loss‚1.5154721523398784 1loss/SRqqt4/detection_loss‚1.6663583221810836 /loss/FBt2/detection_loss‚2.8501832112669945 epoch‚5 1loss/FBt1/assignment_loss‚0.29776335629668954 .loss/FRt2/detection_loss‚2.103640161892947 2loss/SRqqt3/assignment_loss‚1.3859925196350862 1loss/FBt4/assignment_loss‚0.43967680829667766 .loss/FBt4/detection_loss‚2.699319915337996 trainer/global_step‚7199 0loss/FRt3/assignment_loss‚2.6887474603512707 2loss/SRqqt1/assignment_loss‚2.2986659918512617 1loss/FBt2/assignment_loss‚0.33776704425963544 1loss/SRqqt3/detection_loss‚1.6988133278985817 _runtime‚5343.026354305 _step‚590 2loss/SRqqt2/assignment_loss‚1.3066037787096714 /loss/FRt1/detection_loss‚2.8720420466528997 0loss/FRt4/assignment_loss‚3.3510139638727363 /loss/FRt3/detection_loss‚2.3918334486729957 .loss/FRt4/detection_loss‚2.155359273607081 .loss/FBt1/detection_loss‚3.045016960664229֫Ƙ_©Ù /loss/FRt3/detection_loss‚2.3918334486729957 .loss/FRt4/detection_loss‚2.155359273607081 .loss/FBt1/detection_loss‚3.045016960664229 $ _timestamp‚1.788412732613993e+09 %loss/total_loss‚47.65461220993972 /loss/FRt1/assignment_loss‚4.871151685714722 /loss/FRt2/assignment_loss‚2.510581812437843 0loss/SRqqt1/detection_loss‚2.399276981548387 /loss/FBt3/detection_loss‚0.9891267180273438 2loss/SRqqt4/assignment_loss‚1.5911982493406092 0loss/FBt3/assignment_loss‚0.1890070190831025 1loss/SRqqt2/detection_loss‚1.5154721523398784 1loss/SRqqt4/detection_loss‚1.6663583221810836 /loss/FBt2/detection_loss‚2.8501832112669945 epoch‚5 1loss/FBt1/assignment_loss‚0.29776335629668954 .loss/FRt2/detection_loss‚2.103640161892947 2loss/SRqqt3/assignment_loss‚1.3859925196350862 1loss/FBt4/assignment_loss‚0.43967680829667766 .loss/FBt4/detection_loss‚2.699319915337996 trainer/global_step‚7199 0loss/FRt3/assignment_loss‚2.6887474603512707 2loss/SRqqt1/assignment_loss‚2.2986659918512617 1loss/FBt2/assignment_loss‚0.33776704425963544 1loss/SRqqt3/detection_loss‚1.6988133278985817 _runtime‚5343.026354305 _step‚590 2loss/SRqqt2/assignment_loss‚1.3066037787096714 /loss/FRt1/detection_loss‚2.8720420466528997 0loss/FRt4/assignment_loss‚3.3510139638727363ÎâýÈXª‚(:… ȆäÔÿþ¦™ gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.891' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.1782 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ¡-ú «‚(:• ȆäÔé¿èœ disk.nvme0c0n1.in‚ 710.40625' disk.nvme0c0n1.out‚64927.8291015625 disk.sda3.out‚ 213.921875 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda.out‚ 213.921875 disk.rbd1.out‚ 2.822265625 disk.dm-0.in‚ 710.40625% disk./.usageGB‚1098.0210227966309 disk.rbd2.out‚ 123.349609375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0$ memory_percent‚98.47981929779053 cpu‚25.34557607751281 proc.cpu.threads‚17) disk./.usagePercent‚73.68911214088492 disk.rbd0.in‚0" disk.dm-0.out‚64927.8291015625( proc.memory.availableMB‚ 249.06640625 disk.dm-1.out‚ 213.921875 network.sent‚ 1.6665744e+07 network.recv‚ 1.974659e+06 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 710.40625# proc.memory.rssMB‚ 2878.38671875* proc.memory.percent‚17.568278312683105 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0% disk.nvme0n1.out‚64927.8291015625Lxª$‘¬‚(:† ׆äÔÛ¾ý˜ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.756' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 W"­‚(:— ׆äÔäÀºœ disk.dm-0.in‚ 710.40625 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.nvme0n1.in‚ 710.40625 disk.dm-1.out‚ 214.55859375& disk.nvme0c0n1.out‚64928.693359375 disk.sda2.in‚0# proc.memory.rssMB‚ 2878.38671875 cpu‚25.34471131849277 network.sent‚ 1.6773746e+07( disk./.usagePercent‚73.6891157249904 disk.sda.out‚ 214.55859375 disk.drbd1777.out‚0! disk.dm-0.out‚64928.693359375 disk.sda3.out‚ 214.55859375 disk.rbd0.in‚0$ memory_percent‚98.48239421844482$ disk.nvme0n1.out‚64928.693359375 disk.rbd2.out‚ 123.490234375 disk.drbd1777.in‚0 disk.rbd1.out‚ 2.822265625 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.out‚0* proc.memory.percent‚17.568278312683105 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 248.64453125 network.recv‚ 1.980299e+06% disk./.usageGB‚1098.0210762023926 disk.sda.in‚0 disk.sda1.out‚0U+Â2®- _wandbruntime‚5377  _runtime‚5377 /µ¬É¯Ã trainer/global_step‚7249 % _timestamp‚1.7884127665883157e+09 _runtime‚5377.000444559 _step‚591 ' lr-AdamW/pg1‚4.1816964053416647e-05 ' lr-AdamW/pg2‚4.1816964053416647e-05 Gx5Î°È ' lr-AdamW/pg1‚4.1816964053416647e-05 ' lr-AdamW/pg2‚4.1816964053416647e-05 trainer/global_step‚7249 % _timestamp‚1.7884127665883157e+09 _runtime‚5377.000444559 _step‚591ÏÍ[5á2±- _wandbruntime‚5377  _runtime‚5377C+•˜ ²áO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206390272 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚193736938304512 trainer/global_step‚7249 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95160795 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55538956 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10182260702066 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚226194025671680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚193736921265152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚215805210817536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚215805186877440 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚203919364127982 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96544201 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95160795 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213869742831104 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10211669766656 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚203658073064448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213869742831104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚203658073064448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96544723 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146294522 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚5377.001641394 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 % _timestamp‚1.7884127665891528e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 241435279 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95160797 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230330368^µì'³æO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚5377.001641394 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 % _timestamp‚1.7884127665891528e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 241435279 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95160797 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230330368 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206390272 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚193736938304512 trainer/global_step‚7249 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95160795 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55538956 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10182260702066 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚226194025671680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚193736921265152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚215805210817536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚215805186877440 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚203919364127982 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96544201 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95160795 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213869742831104 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10211669766656 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚203658073064448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213869742831104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚203658073064448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96544723 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146294522 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ÐQž>2´- _wandbruntime‚5377  _runtime‚5377µ#…0ð&µêM FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788412767234589e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146314335 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146314335 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55546333 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95173456 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 241467725 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 146294271 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚213898405334016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241487791 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55546331 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10390253261312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚203946696475070 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 trainer/global_step‚7249 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95173454 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚203685342429184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41011179 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 _runtime‚5377.647338135 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚213898592350208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚215834099825664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚215834123765760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚193762864447488 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96556970 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚203685360432128 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10213062904832 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚226224589695488 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10390465929728 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10183814988222 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10183649866610 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95173456 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚593 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 241487791 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10213231918080 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146294271 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10213231918080 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚203685342429184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚203946514314098m–5€¶ïM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146314335 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146314335 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55546333 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95173456 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 241467725 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 146294271 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚213898405334016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241487791 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55546331 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10390253261312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚203946696475070 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 trainer/global_step‚7249 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95173454 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚203685342429184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41011179 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 _runtime‚5377.647338135 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚213898592350208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚215834099825664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚215834123765760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚193762864447488 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J;DeviceStatsMonitor.on_train_batch_enUbŠud/active_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96556970 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚203685360432128 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10213062904832 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚226224589695488 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10390465929728 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10183814988222 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10183649866610 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95173456 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚593 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 241487791 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10213231918080 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146294271 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10213231918080 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚203685342429184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚203946514314098 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788412767234589e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199Ñ%µt2·- _wandbruntime‚5377  _runtime‚5377«üR^¸Ø 0loss/FRt4/assignment_loss‚3.9431131814534846 /loss/FRt3/detection_loss‚1.8774348312410816 /loss/FBt1/detection_loss‚3.2206509330055932 0loss/FBt1/assignment_loss‚0.3419377886961725 0loss/SRqqt2/detection_loss‚1.634797086844505 .loss/FBt2/detection_loss‚4.349242885907491 epoch‚5 1loss/SRqqt4/detection_loss‚2.0540913110193997 /loss/FBt4/detection_loss‚2.4159072807856967 trainer/global_step‚7249 $ _timestamp‚1.788412767236623e+09 0loss/FRt3/assignment_loss‚1.8474723223982186 2loss/SRqqt1/assignment_loss‚2.3715458615072844 /loss/FBt3/detection_loss‚0.8191117279233595 _runtime‚5377.648698348 2loss/SRqqt3/assignment_loss‚1.1843158055286287 2loss/SRqqt4/assignment_loss‚2.0504967779245065 /loss/FRt4/detection_loss‚2.1409333548866787 1loss/SRqqt1/detection_loss‚2.6433425833439004 /loss/FRt1/assignment_loss‚5.131149181297848 2loss/SRqqt2/assignment_loss‚1.8362317051170236 .loss/FRt1/detection_loss‚3.053180766957147 /loss/FRt2/detection_loss‚2.5551724009215833 %loss/total_loss‚50.27334717248219 0loss/FRt2/assignment_loss‚2.8310746759176255 2loss/FBt2/assignment_loss‚0.049036994751077145 1loss/FBt3/assignment_loss‚0.20433623069054646 1loss/FBt4/assignment_loss‚0.41471507493263615 1loss/SRqqt3/detection_loss‚1.3040564094307059 _step‚594±+Wc¹Ý 2loss/FBt2/assignment_loss‚0.049036994751077145 1loss/FBt3/assignment_loss‚0.20433623069054646 1loss/FBt4/assignment_loss‚0.41471507493263615 1loss/SRqqt3/detection_loss‚1.3040564094307059 _step‚594 0loss/FRt4/assignment_loss‚3.9431131814534846 /loss/FRt3/detection_loss‚1.8774348312410816 /loss/FBt1/detection_loss‚3.2206509330055932 0loss/FBt1/assignment_loss‚0.3419377886961725 0loss/SRqqt2/detection_loss‚1.634797086844505 .loss/FBt2/detection_loss‚4.349242885907491 epoch‚5 1loss/SRqqt4/detection_loss‚2.0540913110193997 /loss/FBt4/detection_loss‚2.4159072807856967 trainer/global_step‚7249 $ _timestamp‚1.788412767236623e+09 0loss/FRt3/assignment_loss‚1.8474723223982186 2loss/SRqqt1/assignment_loss‚2.3715458615072844 /loss/FBt3/detection_loss‚0.8191117279233595 _runtime‚5377.648698348 2loss/SRqqt3/assignment_loss‚1.1843158055286287 2loss/SRqqt4/assignment_loss‚2.0504967779245065 /loss/FRt4/detection_loss‚2.1409333548866787 1loss/SRqqt1/detection_loss‚2.6433425833439004 /loss/FRt1/assignment_loss‚5.131149181297848 2loss/SRqqt2/assignment_loss‚1.8362317051170236 .loss/FRt1/detection_loss‚3.053180766957147 /loss/FRt2/detection_loss‚2.5551724009215833 %loss/total_loss‚50.27334717248219 0loss/FRt2/assignment_loss‚2.8310746759176255Ò¦´yɺ‚(:… æ†äÔ²¼ê™ gpu.0.gpu‚18.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0⻂(:Œ æ†äÔ¶Úä›) disk./.usagePercent‚73.68912468525409 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 215.015625 cpu‚25.348568167657664# disk.nvme0n1.out‚65028.38671875 disk.sda.in‚0 disk.sda.out‚ 215.015625 disk.rbd3.out‚0 disk.sda3.out‚ 215.015625# proc.memory.rssMB‚ 2878.38671875 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 710.40625# memory_percent‚98.4823226928711 proc.cpu.threads‚17 disk.rbd0.out‚0% disk.nvme0c0n1.out‚65028.38671875 disk.nvme0n1.in‚ 710.40625 disk.drbd1777.out‚0 disk.dm-0.out‚65028.38671875% proc.memory.availableMB‚ 248.65625* proc.memory.percent‚17.568278312683105% disk./.usageGB‚1098.0212097167969 disk.rbd3.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 2.822265625 disk.dm-0.in‚ 710.40625 disk.rbd2.in‚0 disk.rbd2.out‚ 125.064453125 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 network.sent‚ 1.6779983e+07 network.recv‚ 1.985297e+06É û…¼‚(:„ õ†äÔˆÕ–™ gpu.0.gpu‚5.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.628' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7256 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04“i$#½‚(:˜ õ†äÔòâÀ› disk.sda1.out‚0 disk.rbd2.out‚ 125.267578125 disk.sda3.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.nvme0n1.in‚ 710.40625 disk.dm-1.in‚0 disk.dm-1.out‚ 215.5703125$ memory_percent‚98.48132133483887 disk.sda.out‚ 215.5703125 disk.drbd1777.out‚0 disk.rbd1.out‚ 2.822265625 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 215.5703125# proc.memory.rssMB‚ 2878.38671875) disk./.usagePercent‚73.68931720291985% disk./.usageGB‚1098.0240783691406 disk.drbd1777.in‚0# disk.dm-0.out‚65128.67919921875 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 710.40625' proc.memory.availableMB‚ 248.8203125 network.sent‚ 1.6888594e+07 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0 cpu‚25.3614329360036 disk.sda.in‚0& disk.nvme0n1.out‚65128.67919921875 disk.sda2.in‚0* proc.memory.percent‚17.568278312683105( disk.nvme0c0n1.out‚65128.67919921875 network.recv‚ 1.99167e+06 disk.dm-0.in‚ 710.40625?B\‘¾‚(:† „‡äÔìÏœ™ gpu.0.gpu‚11.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚120.962' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1924 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂAþr¿‚(:Ž „‡äÔœí÷›$ disk./.usageGB‚1098.024013519287 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 710.40625$ disk.nvme0c0n1.out‚ 65230.8984375 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 215.8515625 disk.drbd1777.out‚0 disk.dm-0.out‚ 65230.8984375 disk.rbd0.out‚0 disk.drbd1777.in‚0 disk.dm-1.out‚ 215.8515625 disk.sda2.in‚0$ memory_percent‚98.48010540008545 cpu‚25.373989992834687 network.recv‚ 1.99522e+06" disk.nvme0n1.out‚ 65230.8984375 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 125.267578125 disk.rbd3.out‚0# proc.memory.rssMB‚ 2878.38671875 disk.rbd1.out‚ 2.822265625 disk.nvme0n1.in‚ 710.40625 disk.sda2.out‚0( proc.memory.availableMB‚ 249.01953125* proc.memory.percent‚17.568278312683105 network.sent‚ 1.6893901e+07) disk./.usagePercent‚73.68931285079177 disk.sda1.out‚0 disk.dm-0.in‚ 710.40625 disk.rbd2.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.sda3.out‚ 215.8515625Š6rƒ2À- _wandbruntime‚5427  _runtime‚5427 8®ÞM*ÁÇT dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚10235925281792 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 127308288 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚41263809 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚10206295082306 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚217 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 242503773 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 242487794 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1006 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚199 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 406789969 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚214954757276672 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 146738816 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 4757245257 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚214954611965440 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 146754793 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚416 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 417333248 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚216910326331392 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚204950108388674 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 242487794 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 127308288 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚194743830345728 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚95748978 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚95748980 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚217 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚286 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚55894341 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 410712064 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚227323509514752 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚204950249353626 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 123925592 _step‚595 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚204718831994880 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚284 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚227323649416192 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚416 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 242503773 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚416 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚199 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚95748978 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚97157864 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚10206419007898 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 5041553408 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 5041553408 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 417333248 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚10235925281792 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 115961344 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚214954611965440 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚55894339 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚985 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚214954757276672 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 145311232 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚10235797973504 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚97158150 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 trainer/global_step‚7253 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚204718813991936 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚204718813991936 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0Ô–gM»'ÂÌT E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚416 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚199 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚95748978 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚97157864 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚10206419007898 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 5041553408 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 5041553408 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 417333248 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚10235925281792 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 115961344 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚214954611965440 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚55894339 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚985 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚214954757276672 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 145311232 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚10235797973504 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚97158150 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 trainer/global_step‚7253 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚204718813991936 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚204718813991936 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚10235925281792 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 127308288 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚41263809 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚10206295082306 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚217 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 242503773 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 242487794 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1006 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚199 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 406789969 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚214954757276672 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 146738816 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 4757245257 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚214954611965440 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 146754793 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚416 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 417333248 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚216910326331392 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚204950108388674 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 242487794 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 127308288 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚194743830345728 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚95748978 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚95748980 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚217 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚286 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚55894341 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 410712064 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚227323509514752 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚204950249353626 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 123925592 _step‚595 eQDeviceStatsMonitor.on_validation쀗_batch_start/allocated_bytes.large_pool.allocated‚204718831994880 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚284 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚227323649416192 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚416 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 242503773 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚199ÓN,Ë%2Ã- _wandbruntime‚5428  _runtime‚5428 ¨)úX)ÄÒR NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚194761714860032 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚41268537 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚217 trainer/global_step‚7253 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 242505902 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚217 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 242521881 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 269877760 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚95759692 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 146762189 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 123925592 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 4541568000 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚416 % _timestamp‚1.7884128178991945e+09 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚10236147109376 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚26 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 127308288 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚227343462216192 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚204968498513626 _step‚596 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚214973826287104 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 4912020992 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚216929929896960 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 5041553408 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚214973971598336 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚10236274417664 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚97169378 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚284 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚985 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚199 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚199 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚10413648280576 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 406789969 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 410712064 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 5041553408 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚204737697180672 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚227343602117632 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚216929953837056 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚55900841 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 140964952 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 4757245257 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚416 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚10206642688642 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 5041553408 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 146762189 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚55900839 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚204737697180672 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 127308288 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 410712064}ù9…])Å×R G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚216929929896960 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 5041553408 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚214973971598336 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚10236274417664 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚97169378 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚284 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚985 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚199 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚199 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚10413648280576 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 406789969 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 410712064 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 5041553408 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚204737697180672 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚227343602117632 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚216929953837056 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚55900841 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 140964952 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 4757245257 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚416 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚10206642688642 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 5041553408 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 146762189 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚55900839 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚204737697180672 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 127308288 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 410712064 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚194761714860032 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚41268537 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚217 trainer/global_step‚7253 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 242505902 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚217 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 242521881 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 269877760 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚95759692 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 146762189 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 123925592 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 4541568000 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚416 % _timestamp‚1.7884128178991945e+09 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚10236147109376 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚26 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 127308288 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚227343462216192 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚204968498513626 _step‚596 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚214973826287104 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 4912020992Ôk~šÆ‚(: “‡äÔˆóŽ™ gpu.0.gpu‚4.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.126' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.625200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—¿:™Ç‚(:ˆ “‡äÔíþ«œ) disk./.usagePercent‚73.68931438683697 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 710.40625 disk.dm-1.out‚ 216.40625 network.recv‚ 1.999024e+06 disk.rbd1.out‚ 2.822265625 disk.rbd3.out‚0 disk.sda2.in‚0$ memory_percent‚98.48182201385498 disk.sda.in‚0 disk.sda.out‚ 216.40625 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd2.in‚0# disk.nvme0c0n1.out‚ 65232.109375 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 710.40625 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.dm-0.in‚ 710.40625 disk.sda3.out‚ 216.40625 cpu‚25.38812835109371 network.sent‚ 1.6899387e+07% disk./.usageGB‚1098.0240364074707! disk.nvme0n1.out‚ 65232.109375 disk.dm-0.out‚ 65232.109375! disk.rbd2.out‚125.36572265625 disk.sda3.in‚0( proc.memory.availableMB‚ 248.73828125# proc.memory.rssMB‚ 2878.38671875 disk.sda1.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.568278312683105äÊÒý2È- _wandbruntime‚5440  _runtime‚5440€¨`k$ɦ & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.48717954754829407 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06546562165021896 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.38673558831214905 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.33802083134651184 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08198180794715881 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4572409689426422 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4900297224521637 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9807435870170593 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4922681450843811 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19902662932872772 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5350340604782104 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3163626492023468 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.42714211344718933 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13723228871822357 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 % _timestamp‚1.7884128303106408e+09 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.997589111328125 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.502991795539856 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.236111119389534 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.35343560576438904 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4285714328289032 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 +jet/accuracy_3_of_3‚0.01857057772576809 +jet/accuracy_3_of_6‚0.17307692766189575 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5032744407653809 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4442407488822937 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4791666567325592 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9525861144065857 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3611111044883728 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.37061405181884766 (particle/f_score‚0.37823522090911865 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009745229966938496 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1841953992843628 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_1_of_7‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9910432696342468 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19169645011425018 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4259258806705475 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9670138955116272 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.23103447258472443 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 trainer/global_step‚7253 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.992393970489502 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11419752240180969 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3391975462436676 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8240816593170166 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5745933651924133 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4704626798629761 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06253818422555923 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 particle/accuracy_1_of_5‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.29629626870155334 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.019581837579607964 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 *jet/accuracy_3_of_4‚0.0478566512465477 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14582368731498718 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.43127506971359253 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10039681196212769 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3430555760860443 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.30000001192092896 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 jet/accuracy_4_of_6‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_3‚1 particle/accuracy_2_of_5‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151560306549 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 *jet/accuracy_1_of_1‚0.4658048748970032 particle/accuracy_3_of_5‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5189265608787537 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16450722515583038 particle/accuracy_5_of_7‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7"Purity/*FRt*SRqqt1FBt/event_purity‚0.40176922082901 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11836104840040207 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.17956022918224335 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9649578332901001 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5059083104133606 ,&Puri"Æñ©nty/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4201946258544922 +"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9160757064819336 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9940325617790222 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4564622938632965 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5172618627548218 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0032051282469183207 particle/accuracy_1_of_3‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5252116322517395 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3601365089416504 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4861111044883728 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 particle/accuracy_3_of_6‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 _runtime‚5440.725284639 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4868723452091217 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5265530347824097 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.47607681155204773 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14015087485313416 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17987851798534393 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5664682984352112 *jet/accuracy_2_of_4‚0.3380779027938843 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9885456562042236 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10602529346942902 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.10000000149011612 particle/accuracy_6_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6763697862625122 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20717813074588776 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.026896920055150986 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_3_of_7‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03845188021659851 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.488584041595459 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.32648807764053345 particle/accuracy_1_of_6‚1 jet/accuracy_6_of_7‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1812210977077484 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02358490601181984 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14156869053840637 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.380475252866745 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3672480583190918 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9519895315170288 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4791666567325592 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9930029511451721 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,particle/sensitivity‚0.29394617676734924 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9948342442512512 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9934807419776917 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22066862881183624 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.37748339772224426 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.0626785159111023 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1994047611951828 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1993938386440277 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10283686965703964 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5196545720100403 +jet/accuracy_3_of_5‚0.12805940210819244 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5208333134651184 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03879310190677643 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1504056602716446 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 <"Purity/2FRt*SRqqt2FBt/event_purity‚0.0058139534667134285 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4409722685813904 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.17443624138832092 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9749455451965332 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3536883592605591 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1269841194152832 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5184204578399658 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9855158925056458 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 +"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4260677397251129 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4350769817829132 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.011810281313955784 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.021995464339852333 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.45649173855781555 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13489940762519836 particle/accuracy_3_of_3‚1 particle/accuracy_6_of_6‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.20681819319725037 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.9615384340286255 +jet/accuracy_2_of_6‚0.49358975887298584 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.49845677614212036 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4729575216770172 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9886777997016907 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005956470035016537 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 jet/accuracy_6_of_6‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4729575216770172 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.470743864774704 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_4‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_7_of_7‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.3684210479259491 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9880452752113342 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.09375 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9871484637260437 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_3_of_7‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9897551536560059 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03027024306356907 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2269710749387741 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5064311623573303 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1366046518087387 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4780612587928772 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3845316171646118 jet/accuracy_7_of_7‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.375 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5063153505325317 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 particle/accuracy_5_of_6‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9963726997375488 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.44599395990371704 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.38350775837898254 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2373359203338623 *jet/accuracy_1_of_2‚0.7325336337089539 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.15432099997997284 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1366206258535385 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.49568966031074524 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 _step‚597 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3162767291069031 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14015087485313416 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1956384927034378 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06578947603702545 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9834067821502686 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.00657894741743803 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9874086380004883 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.46415621042251587 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16042256355285645 *jet/accuracy_1_of_5‚0.9327636957168579 particle/accuracy_2_of_7‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1194760873913765 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9862797260284424 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5526315569877625 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.013594469986855984 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4509675204753876 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994527101516724 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3037037253379822 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13290508091449738 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,jet/accuracy_4_of_5‚0.004704301245510578 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16994506120681763 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13406987488269806 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5225614905357361 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9908046126365662 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.47217297554016113 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49000000953674316 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24961192905902863 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.48903676867485046 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.508999228477478 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.35606062412261963 particle/accuracy_2_of_6‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_7‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15199477970600128 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1700197458267212 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6363636255264282 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.978014349937439 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4520833492279053 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.40585586428642273 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5473904013633728 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5342774987220764 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9818713665008545 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3123854696750641 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.20923423767089844 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3898138701915741 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 particle/accuracy_2_of_4‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.36977848410606384 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10190325230360031 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4670977294445038 particle/accuracy_4_of_7‚1 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1413717120885849 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025814268738031387 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18578454852104187 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.29772651195526123 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6397191882133484 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2280370146036148 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3874727487564087 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2725050151348114 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9900366067886353 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.38055554032325745 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5405188798904419 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.28785452246665955 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6363636255264282 *jet/accuracy_2_of_5‚0.4790579080581665 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.236111119389534 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4378545880317688 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 jet/accuracy_4_of_7‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 epoch‚5 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 *jet/accuracy_1_of_4‚0.8928015828132629 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2983357608318329 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994527101516724 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14555826783180237 particle/accuracy_4_of_5‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1921052634716034 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.47999998927116394 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.11973179876804352 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.40571513772010803 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996220469474792 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 jet/accuracy_2_of_7‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4645822048187256 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2265932559967041 8"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.388552188873291 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_7‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.017523305490612984 (particle/accuracy‚0.8492475748062134 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9818366765975952 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014829494059085846 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2675926089286804 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16352960467338562 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.38235294818878174 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04434523731470108 jet/accuracy_5_of_5‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.22474101185798645 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5161592364311218 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23745885491371155 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07124490290880203 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1449451893568039 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13334831595420837 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 )loss/val_total_loss‚48.56902039236081 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.49334385991096497 *jet/accuracy_1_of_3‚0.8439943194389343 ("Purity/*FRt*SRqqt4FBt/event_purity‚0Ý Ç{IÊ’¦ 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4520833492279053 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.40585586428642273 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5473904013633728 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5342774987220764 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9818713665008545 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3123854696750641 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.20923423767089844 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3898138701915741 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 particle/accuracy_2_of_4‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.36977848410606384 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10190325230360031 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4670977294445038 particle/accuracy_4_of_7‚1 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1413717120885849 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025814268738031387 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18578454852104187 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.29772651195526123 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6397191882133484 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2280370146036148 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3874727487564087 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2725050151348114 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9900366067886353 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.38055554032325745 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5405188798904419 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.28785452246665955 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6363636255264282 *jet/accuracy_2_of_5‚0.4790579080581665 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.236111119389534 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4378545880317688 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 jet/accuracy_4_of_7‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 epoch‚5 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FW¹ÉùBt_purity‚0.9351851940155029 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 *jet/accuracy_1_of_4‚0.8928015828132629 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2983357608318329 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994527101516724 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14555826783180237 particle/accuracy_4_of_5‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1921052634716034 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.47999998927116394 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.11973179876804352 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.40571513772010803 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996220469474792 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 jet/accuracy_2_of_7‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4645822048187256 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2265932559967041 8"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.388552188873291 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_7‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.017523305490612984 (particle/accuracy‚0.8492475748062134 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9818366765975952 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014829494059085846 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2675926089286804 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16352960467338562 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.38235294818878174 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04434523731470108 jet/accuracy_5_of_5‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.22474101185798645 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5161592364311218 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23745885491371155 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07124490290880203 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1449451893568039 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13334831595420837 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 )loss/val_total_loss‚48.56902039236081 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.49334385991096497 *jet/accuracy_1_of_3‚0.8439943194389343 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.48717954754829407 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06546562165021896 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.38673558831214905 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.33802083134651184 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08198180794715881 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4572409689426422 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4900297224521637 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9807435870170593 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4922681450843811 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19902662932872772 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5350340604782104 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3163626492023468 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.42714211344718933 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13723228871822357 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 % _timestamp‚1.7884128303106408e+09 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.997589111328125 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.502991795539856 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.236111119389534 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.35343560576438904 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4285714328289032 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 +jet/accuracy_3_of_3‚0.01857057772576809 +jet/accuracy_3_of_6‚0.17307692766189575 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5032744407653809 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4442407488822937 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4791666567325592 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9525861144065857 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3611111044883728 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.37061405181884766 (particle/f_score‚0.37823522090911865 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009745229966938496 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1841953992843628 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_1_of_7‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9910432696342468 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19169645011425018 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4259258806705475 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9670138955116272 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.23103447258472443 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 trainer/global_step‚7253 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.992393970489502 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11419752240180969 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3391975462436676 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8240816593170166 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5745933651924133 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4704626798629761 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06253818422555923 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 particle/accuracy_1_of_5‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.29629626870155334 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.019581837579607964 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 *jet/accuracy_3_of_4‚0.0478566512465477 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14582368731498718 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.43127506971359253 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10039681196212769 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3430555760860443 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.30000001192092896 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 jet/accuracy_4_of_6‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_3‚1 particle/accuracy_2_of_5‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151560306549 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 *jet/accuracy_1_of_1‚0.4658048748970032 particle/accuracy_3_of_5‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5189265608787537 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16450722515583038 particle/accuracy_5_of_7‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7"Purity/*FRt*SRqqt1FBt/event_purity‚0.40176922082901 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11836104840040207 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.17956022918224335 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9649578332901001 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5059083104133606 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4201946258544922 +"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9160757064819336 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9940325617790222 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4564622938632965 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5172618627548218 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0032051282469183207 particle/accuracy_1_of_3‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5252116322517395 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3601365089416504 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4861111044883728 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 particle/accuracy_3_of_6‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 _runtime‚5440.725284639 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4868723452091217 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5265530347824097 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.47607681155204773 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14015087485313416 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17987851798534393 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5664682984352112 *jet/accuracy_2_of_4‚0.3380779027938843 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9885456562042236 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10602529346942902 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.10000000149011612 particle/accuracy_6_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6763697862625122 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.20717813074588776 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.026896920055150986 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_3_of_7‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03845188021659851 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.488584041595459 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.32648807764053345 particle/accuracy_1_of_6‚1 jet/accuracy_6_of_7‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1812210977077484 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02358490601181984 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14156869053840637 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.380475252866745 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3672480583190918 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9519895315170288 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4791666567325592 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9930029511451721 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,particle/sensitivity‚0.29394617676734924 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9948342442512512 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9934807419776917 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22066862881183624 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.37748339772224426 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.0626785159111023 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1994047611951828 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1993938386440277 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10283686965703964 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5196545720100403 +jet/accuracy_3_of_5‚0.12805940210819244 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5208333134651184 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03879310190677643 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1504056602716446 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 <"Purity/2FRt*SRqqt2FBt/event_purity‚0.0058139534667134285 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4409722685813904 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.17443624138832092 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9749455451965332 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3536883592605591 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1269841194152832 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5184204578399658 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9855158925056458 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 +"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4260677397251129 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4350769817829132 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.011810281313955784 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.021995464339852333 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.45649173855781555 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13489940762519836 particle/accuracy_3_of_3‚1 particle/accuracy_6_of_6‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.20681819319725037 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.9615384340286255 +jet/accuracy_2_of_6‚0.49358975887298584 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.49845677614212036 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4729575216770172 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9886777997016907 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005956470035016537 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 jet/accuracy_6_of_6‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4729575216770172 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.470743864774704 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_4‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_7_of_7‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.3684210479259491 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9880452752113342 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.09375 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9871484637260437 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_3_of_7‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9897551536560059 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03027024306356907 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2269710749387741 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5064311623573303 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1366046518087387 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4780612587928772 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3845316171646118 jet/accuracy_7_of_7‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.375 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5063153505325317 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 particle/accuracy_5_of_6‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9963726997375488 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.44599395990371704 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.38350775837898254 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2373359203338623 *jet/accuracy_1_of_2‚0.7325336337089539 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.15432099997997284 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1366206258535385 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.49568966031074524 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 _step‚597 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3162767291069031 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14015087485313416 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1956384927034378 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06578947603702545 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9834067821502686 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.00657894741743803 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9874086380004883 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.46415621042251587 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16042256355285645 *jet/accuracy_1_of_5‚0.9327636957168579 particle/accuracy_2_of_7‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1194760873913765 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9862797260284424 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5526315569877625 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.013594469986855984 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4509675204753876 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994527101516724 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3037037253379822 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13290508091449738 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,jet/accuracy_4_of_5‚0.004704301245510578 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16994506120681763 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13406987488269806 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5225614905357361 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9908046126365662 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.47217297554016113 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49000000953674316 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24961192905902863 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.48903676867485046 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.508999228477478 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.35606062412261963 particle/accuracy_2_of_6‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 particle/aýËo}×ccuracy_1_of_7‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15199477970600128 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1700197458267212 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6363636255264282 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.978014349937439ÕÈDeœË‚(:‘ ¢‡äÔ®ü²™ gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚114.594' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.918799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Xþ!#!Ì‚(:– ¢‡äÔùôªœ disk.dm-0.in‚ 710.40625 disk.rbd3.out‚0% disk.nvme0c0n1.out‚65454.18359375 disk.sda.in‚0( proc.memory.availableMB‚ 376.23828125 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda.out‚ 217.04296875 network.sent‚ 1.7007735e+07 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 710.40625% disk./.usageGB‚1098.0246276855469 disk.rbd1.in‚0# disk.nvme0n1.out‚65454.18359375 disk.dm-1.out‚ 217.04296875 disk.sda2.out‚0! disk.rbd2.out‚125.36572265625 disk.drbd1777.in‚0 cpu‚25.39734811379855 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2878.5703125 disk.sda3.out‚ 217.04296875 disk.sda1.in‚0 network.recv‚ 2.005458e+06) disk./.usagePercent‚73.68935406800477 disk.rbd1.out‚ 2.822265625 disk.nvme0n1.in‚ 710.40625 disk.sda3.in‚0$ memory_percent‚97.70362377166748 disk.dm-0.out‚65454.18359375 disk.rbd0.in‚0 disk.drbd1777.out‚0* proc.memory.percent‚17.569398880004883 proc.cpu.threads‚17(¥BÍ‚(:… ±‡äÔðý·™ gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.825' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.565 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~›'΂(:œ ±‡äÔÂÎí› network.recv‚ 2.011921e+06) disk./.usagePercent‚73.68935406800477 disk.dm-1.in‚0' disk.nvme0c0n1.out‚65553.4052734375 disk.sda2.in‚0% disk./.usageGB‚1098.0246276855469 disk.rbd1.out‚ 2.822265625 disk.rbd2.in‚0 disk.sda3.out‚ 217.67578125 cpu‚25.396402623281084 proc.cpu.threads‚17 network.sent‚ 1.7132112e+07" disk.dm-0.out‚65553.4052734375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.out‚0" proc.memory.rssMB‚ 2878.5703125% disk.nvme0n1.out‚65553.4052734375 disk.dm-1.out‚ 217.67578125 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda3.in‚0$ memory_percent‚97.70388603210449* proc.memory.percent‚17.569398880004883 disk.sda.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 710.40625 disk.rbd3.out‚0' proc.memory.availableMB‚ 376.1953125 disk.nvme0n1.in‚ 710.40625 disk.nvme0c0n1.in‚ 710.40625 disk.sda.out‚ 217.67578125 disk.rbd0.in‚0! disk.rbd2.out‚125.50634765625 disk.drbd1777.in‚0õ þ2Ï- _wandbruntime‚5474  _runtime‚5474"byæÈР$ _timestamp‚1.788412864095041e+09 _runtime‚5474.507201483 _step‚598 ' lr-AdamW/pg1‚4.1099516809111444e-05 ' lr-AdamW/pg2‚4.1099516809111444e-05 trainer/global_step‚7299㶖ÍÑÇ _runtime‚5474.507201483 _step‚598 ' lr-AdamW/pg1‚4.1099516809111444e-05 ' lr-AdamW/pg2‚4.1099516809111444e-05 trainer/global_step‚7299 $ _timestamp‚1.788412864095041e+09Ö€Ýø2Ò- _wandbruntime‚5474  _runtime‚5474RŸÐ«ç'ÓáO C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚206210066009088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _step‚599 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96468603 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56317304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚196161895302144 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10273813948424 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96468603 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚216513401646080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41570419 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚502 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚218485858919424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚196161878262784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚206210066009088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚218485882859520 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚216513401646080 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96468601 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10303353639936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚216513588648960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10481855971328 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147755109 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚206210048006144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚216513588648960 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10303522639872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228967714890752 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244203672 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚206210048006144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244223712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚206435527089804 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10273648827020 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _runtime‚5474.508246873 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚206435709250568 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96468601 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56317306 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10482060264448 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408ð37ì'ÔæO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚206210066009088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _step‚599 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96468603 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56317304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚196161895302144 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10273813948424 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96468603 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚216513401646080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41570419 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚502 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚218485858919424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚196161878262784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚206210066009088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚218485882859520 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚216513401646080 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96468601 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10303353639936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚216513588648960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10481855971328 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147755109 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚206210048006144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚216513588648960 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10303522639872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228967714890752 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244203672 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚206210048006144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244223712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚206435527089804 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10273648827020 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _runtime‚5474.508246873 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚206435709250568 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96468601 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56317306 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10482060264448 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416×Ûi ‘‘Õ‚(:† À‡äԧ胚 gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.681' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5362 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QÍ1*Ö‚(:Ÿ À‡äÔÀë œ disk.dm-1.out‚ 217.99609375 disk.sda.in‚0 disk.drbd1777.out‚0( proc.memory.availableMB‚ 375.94921875 disk.rbd1.in‚0 disk.rbd1.out‚ 2.908203125 disk.rbd0.in‚0 disk.sda2.out‚0 cpu‚25.395499403098647 proc.cpu.threads‚17 network.sent‚ 1.7137547e+07 disk.nvme0n1.in‚ 710.40625 disk.rbd3.out‚0 disk.sda3.out‚ 217.99609375 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 710.40625( disk.nvme0c0n1.out‚65677.16064453125) disk./.usagePercent‚73.68932053101778& disk.nvme0n1.out‚65677.16064453125 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2878.5703125* proc.memory.percent‚17.569398880004883$ disk./.usageGB‚1098.024127960205 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda.out‚ 217.99609375# disk.dm-0.out‚65677.16064453125 disk.rbd2.in‚0 network.recv‚ 2.016064e+06! disk.rbd2.out‚125.50634765625 disk.sda3.in‚0 disk.drbd1777.in‚0$ memory_percent‚97.70538806915283 disk.dm-0.in‚ 710.40625ÌO3Õ2×- _wandbruntime‚5475  _runtime‚5475弓ØëM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228998276022784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚206237317370880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚206463041597656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56324683 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚228998041511424 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10275203113176 _step‚600 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244256184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10483269643776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚216542251165184 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10304746778112 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚216542064148992 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚7299 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147754858 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10304915791360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96481262 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚206462859436684 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 % _timestamp‚1.7884128647529774e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10304915791360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚196187838484480 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244256184 H@DeviceStatsMonitor.on_train_æ,ŒÖ^ batch_end/inactive_split.all.current‚522 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97899810 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96481260 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚216542251165184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10304746778112 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚206237335373824 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147774922 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147754858 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚206237317370880 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚218514795807744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56324681 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96481262 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41575649 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244236118 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚218514771867648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10275037991564¦½ö&ÙðM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147774922 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244236118 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96481260 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10275037991564 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228998276022784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚206237317370880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚206463041597656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56324683 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚228998041511424 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10275203113176 _step‚600 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244256184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10483269643776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚216542251165184 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10304746778112 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚216542064148992 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚7299 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147754858 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10304915791360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96481262 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚206462859436684 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 % _timestamp‚1.7884128647529774e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10304915791360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚196187838484480 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244256184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97899810 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96481260 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚216542251165184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10304746778112 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚206237335373824 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147774922 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147754858 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚206237317370880 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚218514795807744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56324681 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96481262 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41575649 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244236118 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚218514771867648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97101551 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚535 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚207573534245888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10552934021120 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97101553 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41840805 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚5509.334267683 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚537 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚207802340185776 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚219931530270720 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10373193962496 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148747811 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚217946541205504 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10373024962560 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚217946728208384 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚207573516242944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56686154 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41841340 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚98526959 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚197459054418944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10373024962560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚197459037379584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10343285766832 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 245849364 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199™qö9 ææO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97101551 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148727773 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚207573534245888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97101553 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚7349 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚230484464291840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10552721339392 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚207573516242944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10343120645428 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚98527496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10373193962496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148727773 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚217946541205504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97101551 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚535 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚207573534245888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10552934021120 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97101553 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41840805 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚5509.334267683 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚537 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚207802340185776 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚219931530270720 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10373193962496 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148747811 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚217946541205504 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10373024962560 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚217946728208384 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚207573516242944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56686154 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41841340 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚98526959 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚197459054418944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10373024962560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚197459037379584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10343285766832 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 245849364 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚219960443218944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚217975203708416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41846049 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 _step‚604 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 245881836 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 148747560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10344674931584 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚230514565046784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10554145767936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10374587113984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 245861770 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224025600 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148747560 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚207829672532864 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚219960419278848 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10554345853440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚230514789072384 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98539580 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚217975390724608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200085504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10374418100736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10374418100736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97114210 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148767624 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10344509809972 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚207600803610624 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚207600803610624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚98540094 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97114212 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚207600785607680 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚197484997601280 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚207600785607680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚197484980561920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0±¢Åõ&éïM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚230514565046784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10554145767936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10374587113984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 245861770 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224025600 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148747560 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚207829672532864 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚219960419278848 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10554345853440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚230514789072384 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98539580 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚217975390724608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200085504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10374418100736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10374418100736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97114210 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148767624 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10344509809972 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚207600803610624 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚207600803610624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚98540094 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97114212 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚207600785607680 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚197484997601280 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚207600785607680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚197484980561920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 trainer/global_step‚7349 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 $ _timestamp‚1.788412899578757e+09 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚219960443218944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚217975203708416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41846049 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 _step‚604 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 245881836 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 148747560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10344674931584Ü ¶32ê- _wandbruntime‚5509  _runtime‚5509nÏ-=VëÐ /loss/FRt3/assignment_loss‚2.826711038748423 2loss/SRqqt3/assignment_loss‚1.4812811541336555 2loss/SRqqt4/assignment_loss‚1.8157358410889688 0loss/FBt4/assignment_loss‚0.6529407389166408 /loss/FRt2/detection_loss‚2.2618277343334974 0loss/FBt2/assignment_loss‚0.7309276482444277 /loss/FRt1/detection_loss‚2.6539223056573134 trainer/global_step‚7349 _runtime‚5509.992886952 % _timestamp‚1.7884128995808437e+09 /loss/FRt1/assignment_loss‚4.642413084323589 2loss/SRqqt2/assignment_loss‚1.2427643723785877 .loss/FRt4/detection_loss‚2.132762189241166 0loss/SRqqt1/detection_loss‚2.495898886160417 1loss/SRqqt2/detection_loss‚1.5336338557799658 .loss/FBt4/detection_loss‚3.052755881100893 _step‚605 0loss/FBt1/assignment_loss‚0.4962995553586162 /loss/FRt3/detection_loss‚1.9598061174509072 0loss/SRqqt3/detection_loss‚1.546651292178366 /loss/FBt1/detection_loss‚2.8449969674859727 /loss/FBt3/detection_loss‚0.7464823274123217 1loss/SRqqt1/assignment_loss‚2.032122085830479 0loss/FBt3/assignment_loss‚0.3729456593350354 .loss/FBt2/detection_loss‚3.517726310661861 %loss/total_loss‚49.28817195996199 epoch‚6 1loss/SRqqt4/detection_loss‚1.8818038889895314 /loss/FRt2/assignment_loss‚2.976000087128745 0loss/FRt4/assignment_loss‚3.3897629380226135MÛ [ìÕ 0loss/FBt4/assignment_loss‚0.6529407389166408 /loss/FRt2/detection_loss‚2.2618277343334974 0loss/FBt2/assignment_loss‚0.7309276482444277 /loss/FRt1/detection_loss‚2.6539223056573134 trainer/global_step‚7349 _runtime‚5509.992886952 % _timestamp‚1.7884128995808437e+09 /loss/FRt1/assignment_loss‚4.642413084323589 2loss/SRqqt2/assignment_loss‚1.2427643723785877 .loss/FRt4/detection_loss‚2.132762189241166 0loss/SRqqt1/detection_loss‚2.495898886160417 1loss/SRqqt2/detection_loss‚1.5336338557799658 .loss/FBt4/detection_loss‚3.052755881100893 _step‚605 0loss/FBt1/assignment_loss‚0.4962995553586162 /loss/FRt3/detection_loss‚1.9598061174509072 0loss/SRqqt3/detection_loss‚1.546651292178366 /loss/FBt1/detection_loss‚2.8449969674859727 /loss/FBt3/detection_loss‚0.7464823274123217 1loss/SRqqt1/assignment_loss‚2.032122085830479 0loss/FBt3/assignment_loss‚0.3729456593350354 .loss/FBt2/detection_loss‚3.517726310661861 %loss/total_loss‚49.28817195996199 epoch‚6 1loss/SRqqt4/detection_loss‚1.8818038889895314 /loss/FRt2/assignment_loss‚2.976000087128745 0loss/FRt4/assignment_loss‚3.3897629380226135 /loss/FRt3/assignment_loss‚2.826711038748423 2loss/SRqqt3/assignment_loss‚1.4812811541336555 2loss/SRqqt4/assignment_loss‚1.8157358410889688ÝÎì í‚(:„ í‡äÔÛ®¹™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.63' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T[×2$î‚(:™ í‡äÔ¸€ßœ disk.rbd2.out‚127.2451171875 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 710.40625* proc.memory.percent‚17.569398880004883 cpu‚25.39264558207854 network.recv‚ 2.032631e+06% disk./.usageGB‚1098.0243110656738 disk.sda3.in‚0! disk.dm-0.out‚65980.712890625 disk.rbd0.out‚0$ memory_percent‚97.71325588226318 network.sent‚ 1.7338575e+07 disk.sda.in‚0 disk.sda.out‚ 222.57421875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd3.in‚0) disk./.usagePercent‚73.68933281937944 disk.drbd1777.out‚0" proc.memory.rssMB‚ 2878.5703125 disk.sda2.out‚0 disk.rbd1.out‚ 3.0029296875 disk.dm-0.in‚ 710.40625 disk.dm-1.out‚ 222.57421875& disk.nvme0c0n1.out‚65980.712890625( proc.memory.availableMB‚ 374.66015625 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda3.out‚ 222.57421875$ disk.nvme0n1.out‚65980.712890625 disk.nvme0n1.in‚ 710.40625ßÞn@›ï‚(: ü‡äԚ˙ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.586' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.717200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=q½’ð‚(:š ü‡äÔ‚“þ› network.recv‚ 2.038209e+06( disk./.usagePercent‚73.6893489478541 disk.nvme0c0n1.in‚ 710.40625' disk.nvme0c0n1.out‚65984.0498046875 disk.sda3.out‚ 223.26953125 disk.sda3.in‚0 disk.nvme0n1.in‚ 710.40625 disk.sda.out‚ 223.26953125 disk.dm-0.in‚ 710.40625 disk.dm-1.in‚0 disk.rbd0.in‚0 network.sent‚ 1.7446487e+07% disk./.usageGB‚1098.0245513916016 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 3.0029296875* proc.memory.percent‚17.569398880004883% disk.nvme0n1.out‚65984.0498046875 disk.rbd0.out‚0" proc.memory.rssMB‚ 2878.5703125 cpu‚25.39168338740059 disk.rbd3.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 223.26953125 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0# memory_percent‚97.7104902267456 disk.rbd3.in‚0 disk.drbd1777.in‚0 disk.dYÉB“rbd1777.out‚0" disk.dm-0.out‚65984.0498046875 disk.rbd2.in‚0 disk.rbd2.out‚127.2451171875( proc.memory.availableMB‚ 375.11328125¨ø–2ñ- _wandbruntime‚5543  _runtime‚55430€\#Éòà _runtime‚5543.988681986 _step‚606 ' lr-AdamW/pg1‚3.9670659436857724e-05 ' lr-AdamW/pg2‚3.9670659436857724e-05 trainer/global_step‚7399 % _timestamp‚1.7884129335765548e+09ã0üfÎóÈ ' lr-AdamW/pg1‚3.9670659436857724e-05 ' lr-AdamW/pg2‚3.9670659436857724e-05 trainer/global_step‚7399 % _timestamp‚1.7884129335765548e+09 _runtime‚5543.988681986 _step‚606Þ4a=^2ô- _wandbruntime‚5543  _runtime‚5543?bç:ç'õáO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚209168788960220 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚208936984479744 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10623539128320 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 % _timestamp‚1.7884129335772936e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149720475 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚219379680764928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚219379867767808 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97734503 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10442696285184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97734501 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚219379680764928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚208937002482688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚219379867767808 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149740513 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99166481 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57055006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42111475 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚607 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99165930 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42110926 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚198756196496384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10442865285120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚198756213535744 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚7399 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 247454976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57055004 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10412592463836 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99166481 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57055006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42111475 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚607 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99165930 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42110926 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚198756196496384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10442865285120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚198756213535744 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚7399 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 247454976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57055004 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10412592463836 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚219379867767808 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97734503 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10442696285184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97734501 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚219379680764928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚208937002482688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚219379867767808 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408ßp%„k2÷- _wandbruntime‚5544  _runtime‚5544™ºùð&øêM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚219408343267840 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚221406090630144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚198782156718080 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 % _timestamp‚1.7884129342249405e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57062383 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97747162 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247507488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚232031027180544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10444258436608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42116277 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232031259594752 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10413981628380 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚219408530284032 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97747160 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247487422 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚208964253844480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _runtime‚ 5544.63769063 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97747162 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚208964271847424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚541 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99178658 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚198782139678720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 247507488 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57062381 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42116816 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149760326 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10444089423360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10444089423360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚99179199 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10444258436608 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97747160 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149760326 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149740262 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 trainer/global_step‚7399 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10625168964608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10414146749992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149740262 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149740262 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247507488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚232031027180544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10444258436608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42116277 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232031259594752 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10413981628380 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚219408530284032 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97747160 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247487422 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚208964253844480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _runtime‚ 5544.63769063 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97747162 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚208964271847424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚541 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99178658 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚198782139678720 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 247507488 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57062381 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42116816 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149760326 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10444089423360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10444089423360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚99179199 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10444258436608 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97747160 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149760326 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149740262 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 trainer/global_step‚7399 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10625168964608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10414146749992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10512536607744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150733215 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42383331 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚220813007327232 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249100668 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10482064282244 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 % _timestamp‚1.7884129684710088e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57423854 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10482229403648 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98367453 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150713177 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99807724 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10694646397440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚200053355613184 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98367453 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚210300470719488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42383868 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150733215 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98367451 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249080628 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚222822801153024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚210535602056192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10512536607744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚220812820324352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98367451 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚210300452716544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99807185 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218973184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10512367607808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 trainer/global_step‚7449 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 B²šª†æO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10694427424256 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚233517471490560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚220812820324352 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚210300470719488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚210535419895428 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10512536607744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150733215 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42383331 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚220813007327232 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249100668 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10482064282244 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 % _timestamp‚1.7884129684710088e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57423854 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10482229403648 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98367453 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150713177 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99807724 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10694646397440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚200053355613184 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98367453 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚210300470719488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42383868 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150733215 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98367451 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249080628 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚222822801153024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚210535602056192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10512536607744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚220812820324352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 SDDeviceStatsMonitor.on_train_baŸEïB tch_start/reserved_bytes.all.allocated‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98367451 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚210300452716544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99807185 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218973184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10512367607808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 trainer/global_step‚7449 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚210562934403280 trainer/global_step‚7449 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚210327740084224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10513760745984 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚220841482827264 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150753028 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249133140 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150732964 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚200079315834880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10513929759232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10513760745984 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚220841669843456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚222851714101248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98380110 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚612 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42388727 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚233547556611072 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210562752242308 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚200079298795520 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57431233 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10483618568400 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150732964 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚210327740084224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99819958 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _runtime‚5579.530484879 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150753028 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006¦œÿö&‰ðM ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10513929759232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10513760745984 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚220841669843456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚222851714101248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98380110 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚612 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42388727 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚233547556611072 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210562752242308 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚200079298795520 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57431233 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10483618568400 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150732964 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚210327740084224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99819958 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _runtime‚5579.530484879 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150753028 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 249113074 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚210327722081280 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚210562934403280 trainer/global_step‚7449 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚210327740084224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10513760745984 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚220841482827264 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150753028 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249133140 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150732964 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚200079315834880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000äWV¶2Š- _wandbruntime‚5579  _runtime‚5579;£ÕS‹Í 0loss/SRqqt1/detection_loss‚2.867176892964736 .loss/FBt3/detection_loss‚1.003580088507557 epoch‚6 _runtime‚5579.531968945 /loss/FRt3/assignment_loss‚2.320030949115753 1loss/SRqqt3/assignment_loss‚1.596366594173014 0loss/SRqqt3/detection_loss‚1.468973856717348 1loss/SRqqt4/detection_loss‚1.6711112518431777 .loss/FRt2/assignment_loss‚2.88900674978892 2loss/SRqqt2/assignment_loss‚1.2946485276208133 /loss/FBt2/detection_loss‚2.2181463092565536 1loss/FBt3/assignment_loss‚0.19888818421213728 /loss/FRt3/detection_loss‚1.9113611455261708 .loss/FBt1/detection_loss‚3.116336088627577 .loss/FBt4/detection_loss‚2.388925341459421 trainer/global_step‚7449 _step‚613 0loss/FBt2/assignment_loss‚0.7906625866889954 -loss/FRt1/detection_loss‚3.21498551822844 0loss/SRqqt2/detection_loss‚1.438389450915762 %loss/total_loss‚47.42611309817465 % _timestamp‚1.7884129691198845e+09 0loss/FRt4/assignment_loss‚3.6201768057388173 2loss/SRqqt1/assignment_loss‚1.9735308470933333 2loss/SRqqt4/assignment_loss‚1.9343882649821245 0loss/FBt4/assignment_loss‚0.4107972850966829 /loss/FRt4/detection_loss‚2.0121780994691347 /loss/FRt1/assignment_loss‚4.523671093441191 0loss/FBt1/assignment_loss‚0.3386328031697303 /loss/FRt2/detection_loss‚2.2241483635372585u};XŒÒ 0loss/FBt2/assignment_loss‚0.7906625866889954 -loss/FRt1/detection_loss‚3.21498551822844 0loss/SRqqt2/detection_loss‚1.438389450915762 %loss/total_loss‚47.42611309817465 % _timestamp‚1.7884129691198845e+09 0loss/FRt4/assignment_loss‚3.6201768057388173 2loss/SRqqt1/assignment_loss‚1.9735308470933333 2loss/SRqqt4/assignment_loss‚1.9343882649821245 0loss/FBt4/assignment_loss‚0.4107972850966829 /loss/FRt4/detection_loss‚2.0121780994691347 /loss/FRt1/assignment_loss‚4.523671093441191 0loss/FBt1/assignment_loss‚0.3386328031697303 /loss/FRt2/detection_loss‚2.2241483635372585 0loss/SRqqt1/detection_loss‚2.867176892964736 .loss/FBt3/detection_loss‚1.003580088507557 epoch‚6 _runtime‚5579.531968945 /loss/FRt3/assignment_loss‚2.320030949115753 1loss/SRqqt3/assignment_loss‚1.596366594173014 0loss/SRqqt3/detection_loss‚1.468973856717348 1loss/SRqqt4/detection_loss‚1.6711112518431777 .loss/FRt2/assignment_loss‚2.88900674978892 2loss/SRqqt2/assignment_loss‚1.2946485276208133 /loss/FBt2/detection_loss‚2.2181463092565536 1loss/FBt3/assignment_loss‚0.19888818421213728 /loss/FRt3/detection_loss‚1.9113611455261708 .loss/FBt1/detection_loss‚3.116336088627577 .loss/FBt4/detection_loss‚2.388925341459421 trainer/global_step‚7449 _step‚613åÖa~œ‚(:‘ ©ˆäÔÞ›‘™ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚106.842' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.368399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì%s(Ž‚(: ©ˆäÔ¡†œ disk.rbd2.in‚0$ memory_percent‚98.30045700073242% disk./.usageGB‚1098.0237350463867 disk.sda.in‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0" proc.memory.rssMB‚ 2878.5703125 network.sent‚ 1.7565993e+07 network.recv‚ 2.052435e+06 disk.sda1.out‚0 disk.rbd1.out‚ 3.12890625# disk.dm-0.out‚66281.47998046875 disk.dm-1.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 225.03515625 disk.rbd0.in‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 278.453125) disk./.usagePercent‚73.68929416224177 disk.nvme0n1.in‚ 710.40625! disk.rbd2.out‚129.06201171875 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.out‚ 225.03515625& disk.nvme0n1.out‚66281.47998046875( disk.nvme0c0n1.out‚66281.47998046875 disk.drbd1777.in‚0* proc.memory.percent‚17.569398880004883 disk.sda.out‚ 225.03515625 disk.rbd1.in‚0 disk.dm-0.in‚ 710.40625 cpu‚25.388939414000898 disk.nvme0c0n1.in‚ 710.40625 disk.sda2.in‚0 disk.sda3.in‚0 $\›‚(: ¸ˆäÔء™ gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚94.659' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.931800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n€4$‚(:™ ¸ˆäÔ¢ˆœ disk.sda1.out‚0 disk.rbd1.out‚ 3.12890625 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 710.40625& disk.nvme0c0n1.out‚66282.189453125% disk./.usageGB‚1098.0236587524414 disk.sda1.in‚0 disk.dm-0.in‚ 710.40625 disk.dm-1.out‚ 225.47265625 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0" proc.memory.rssMB‚ 2878.5703125 cpu‚25.388078648338045 proc.cpu.threads‚17 disk.drbd1777.out‚0! disk.dm-0.out‚66282.189453125 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda.out‚ 225.47265625 disk.nvme0n1.in‚ 710.40625! disk.rbd2.out‚129.07763671875 disk.rbd3.out‚0$ memory_percent‚98.46656322479248 disk.sda3.in‚0$ disk.nvme0n1.out‚66282.189453125 disk.rbd0.in‚0* proc.memory.percent‚17.569398880004883 network.recv‚ 2.057947e+06) disk./.usagePercent‚73.68928904209108 disk.rbd3.in‚0( proc.memory.availableMB‚ 251.23828125 network.sent‚ 1.7673855e+07 disk.sda3.out‚ 225.47265625åΑ‚(:„ LjäÔÔ–¶™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.63' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€f$þ*’‚(:Ÿ LjäÔ™¢ê› network.sent‚ 1.7679287e+07 disk.sda3.out‚ 226.05078125" proc.memory.rssMB‚ 2878.5703125* proc.memory.percent‚17.569398880004883 network.recv‚ 2.061821e+06( disk.nvme0c0n1.out‚66381.26318359375 disk.sda.out‚ 226.05078125 disk.rbd1.out‚ 3.12890625 disk.nvme0n1.in‚ 710.40625$ memory_percent‚98.46627712249756 disk.rbd0.out‚0! disk.rbd2.out‚129.14013671875 disk.sda.in‚0 disk.drbd1777.out‚0 disk.drbd1777.in‚0& disk.nvme0n1.out‚66381.26318359375 disk.dm-0.in‚ 710.40625 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0 cpu‚25.387136833179174 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 251.28515625 disk.dm-1.out‚ 226.05078125 disk.nvme0c0n1.in‚ 710.40625 disk.sda1.in‚0) disk./.usagePercent‚73.68928904209108% disk./.usageGB‚1098.0236587524414 disk.dm-1.in‚0 disk.rbd0.in‚0# disk.dm-0.out‚66381.26318359375^0,2“- _wandbruntime‚5613  _runtime‚5613–;¦Æ”À trainer/global_step‚7499 $ _timestamp‚1.788413003170542e+09 _runtime‚5613.582686956 _step‚614 & lr-AdamW/pg1‚3.825084336106211e-05 & lr-AdamW/pg2‚3.825084336106211e-05übÞùË•Å & lr-AdamW/pg1‚3.825084336106211e-05 & lr-AdamW/pg2‚3.825084336106211e-05 trainer/global_step‚7499 $ _timestamp‚1.788413003170542e+09 _runtime‚5613.582686956 _step‚614æ©&ó2–- _wandbruntime‚5613  _runtime‚5613«ïî´ —ãO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99000403 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10582038930432 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10582207930368 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚201350531769344 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10551701222056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42654241 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚222246146886656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57792706 _runtime‚5613.583676961 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 250706280 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151705879 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100447507 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚201350514729984 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 \IDeviceStaètž ÐtsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10582207930368 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151725917 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99000401 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10765507953664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151725917 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚211902050830636 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚211663920953344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99000401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99000403 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 151705879 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚211663938956288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224268472504320 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚211902232991400 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7884130031713135e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218973184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚222246146886656 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 250726320 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚562 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚7499 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚222245959883776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 _step‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10582038930432 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚235033737544704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10765288980480 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42654801 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚211663938956288 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚211663920953344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10551536100652 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57792704 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚224268448564224Ù*³Nî'˜èO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151725917 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99000401 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10765507953664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151725917 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚211902050830636 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚211663920953344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99000401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99000403 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 151705879 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚211663938956288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224268472504320 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚211902232991400 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7884130031713135e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218973184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚222246146886656 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 250726320 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚562 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚7499 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚222245959883776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 _step‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10582038930432 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚235033737544704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10765288980480 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42654801 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚211663938956288 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚211663920953344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10551536100652 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57792704 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚224268448564224 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99000403 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10582038930432 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10582207930368 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚201350531769344 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10551701222056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42654241 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚222246146886656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57792706 _runtime‚5613.583676961 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 250706280 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151705879 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100447507 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚201350514729984 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10582207930368ç6‹72™- _wandbruntime‚5614  _runtime‚5614€°®Jó&šíM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10553090386808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚222274809402880 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚533 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚211691208321024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250738726 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10552925265196 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚224297385452544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100459758 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚211929383177516 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 _step‚616 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99013062 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 trainer/global_step‚7499 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99013060 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235064066734592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10766705222144 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10583601081856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57800083 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250758792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250758792 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚222274809402880 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151725666 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969íéì¡ú›òM trainer/global_step‚7499 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99013060 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235064066734592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10766705222144 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10583601081856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57800083 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250758792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250758792 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚222274809402880 P8DeviceStatsMonitor.on_train_batcÓRÑAþh_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151725666 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10553090386808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚222274809402880 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚533 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚211691208321024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250738726 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10552925265196 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚224297385452544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100459758 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚211929383177516 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 _step‚616 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99013062 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160苯 è2œ- _wandbruntime‚5614  _runtime‚5614ËŠ€PSÍ 2loss/SRqqt3/assignment_loss‚1.4602426074571528 .loss/FRt2/detection_loss‚2.037418158352375 /loss/FBt3/assignment_loss‚0.301519730134789 /loss/FRt4/detection_loss‚2.5491313762962817 1loss/SRqqt3/detection_loss‚1.5234355868424398 .loss/FBt1/detection_loss‚3.767033439129591 _step‚617 2loss/SRqqt4/assignment_loss‚1.5885879022754588 /loss/FBt3/detection_loss‚0.8386986347057843 trainer/global_step‚7499 0loss/FBt4/assignment_loss‚0.3999340814431586 1loss/SRqqt1/detection_loss‚3.1008768746486077 %loss/total_loss‚51.36129972282821 /loss/FRt3/assignment_loss‚2.815547873365118 /loss/FRt4/assignment_loss‚3.573763903975487 1loss/SRqqt1/assignment_loss‚2.624534561083867 2loss/SRqqt2/assignment_loss‚1.4705632709239314 /loss/FRt3/detection_loss‚2.2927486852128456 _runtime‚5614.230830041 1loss/FBt1/assignment_loss‚0.46982526431429505 1loss/SRqqt4/detection_loss‚1.9397780681556125 /loss/FBt2/detection_loss‚2.7786131858825684 -loss/FRt1/detection_loss‚2.99099188297987 0loss/SRqqt2/detection_loss‚1.347519053373395 .loss/FBt4/detection_loss‚2.340046316385269 /loss/FRt1/assignment_loss‚5.150789134204388 /loss/FRt2/assignment_loss‚2.805013922850291 0loss/FBt2/assignment_loss‚1.1946862088356283 epoch‚6 $ _timestamp‚1.788413003818791e+09í\¹XžÒ -loss/FRt1/detection_loss‚2.99099188297987 0loss/SRqqt2/detection_loss‚1.347519053373395 .loss/FBt4/detection_loss‚2.340046316385269 /loss/FRt1/assignment_loss‚5.150789134204388 /loss/FRt2/assignment_loss‚2.805013922850291 0loss/FBt2/assignment_loss‚1.1946862088356283 epoch‚6 $ _timestamp‚1.788413003818791e+09 2loss/SRqqt3/assignment_loss‚1.4602426074571528 .loss/FRt2/detection_loss‚2.037418158352375 /loss/FBt3/assignment_loss‚0.301519730134789 /loss/FRt4/detection_loss‚2.5491313762962817 1loss/SRqqt3/detection_loss‚1.5234355868424398 .loss/FBt1/detection_loss‚3.767033439129591 _step‚617 2loss/SRqqt4/assignment_loss‚1.5885879022754588 /loss/FBt3/detection_loss‚0.8386986347057843 trainer/global_step‚7499 0loss/FBt4/assignment_loss‚0.3999340814431586 1loss/SRqqt1/detection_loss‚3.1008768746486077 %loss/total_loss‚51.36129972282821 /loss/FRt3/assignment_loss‚2.815547873365118 /loss/FRt4/assignment_loss‚3.573763903975487 1loss/SRqqt1/assignment_loss‚2.624534561083867 2loss/SRqqt2/assignment_loss‚1.4705632709239314 /loss/FRt3/detection_loss‚2.2927486852128456 _runtime‚5614.230830041 1loss/FBt1/assignment_loss‚0.46982526431429505 1loss/SRqqt4/detection_loss‚1.9397780681556125 /loss/FBt2/detection_loss‚2.7786131858825684éûà ‘Ÿ‚(:† ÖˆäÔຶ™ gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.224' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2448 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v¹,F$ ‚(:™ ÖˆäÔÐû²œ" disk.dm-0.out‚66481.0537109375 disk.rbd2.in‚0! disk.rbd2.out‚130.72998046875 disk.rbd3.out‚0 disk.drbd1777.out‚0 network.recv‚ 2.066903e+06 disk.rbd1.out‚ 3.12890625 disk.rbd3.in‚0" proc.memory.rssMB‚ 2878.5703125 network.sent‚ 1.7685576e+07 disk.nvme0n1.in‚ 710.40625 disk.dm-1.out‚ 226.45703125 disk.nvme0c0n1.in‚ 710.40625 disk.sda.in‚0 disk.sda1.out‚0% proc.memory.availableMB‚ 250.96875 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚66481.0537109375 disk.sda1.in‚0 disk.rbd1.in‚0 disk.sda3.in‚0 disk.drbd1777.in‚0* proc.memory.percent‚17.569398880004883 disk.dm-1.in‚0 disk.sda2.in‚0% disk.nvme0n1.out‚66481.0537109375 disk.sda.out‚ 226.45703125% disk./.usageGB‚1098.0246353149414 disk.dm-0.in‚ 710.40625 disk.sda2.out‚0 disk.sda3.out‚ 226.45703125$ memory_percent‚98.46820831298828 cpu‚25.386280610482867) disk./.usagePercent‚73.68935458001985Õª‘¡‚(:† åˆäÔ›Ô¬™ gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.266' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0532 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇY³A&¢‚(:› åˆäÔ‡ãâœ' disk.nvme0c0n1.out‚66580.1357421875* proc.memory.percent‚17.569398880004883% disk./.usageGB‚1098.0246353149414 disk.sda2.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚0 cpu‚25.385386740215374 disk.nvme0n1.in‚ 710.40625" proc.memory.rssMB‚ 2878.5703125) disk./.usagePercent‚73.68935458001985 disk.drbd1777.out‚0 disk.dm-0.in‚ 710.40625" disk.dm-0.out‚66580.1357421875 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 710.40625# memory_percent‚98.4684705734253 disk.rbd3.out‚0 network.recv‚ 2.073067e+06 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 227.13671875 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd1777.in‚0% disk.nvme0n1.out‚66580.1357421875 network.sent‚ 1.7794129e+07 disk.sda.out‚ 227.13671875 disk.dm-1.out‚ 227.13671875 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 3.12890625 disk.rbd3.in‚0! disk.rbd2.out‚130.79248046875( proc.memory.availableMB‚ 250.92578125cÈ 2£- _wandbruntime‚5648  _runtime‚5648âI3«Ç¤Á % _timestamp‚1.7884130378963542e+09 _runtime‚5648.308572545 _step‚618 & lr-AdamW/pg1‚3.754469597090507e-05 & lr-AdamW/pg2‚3.754469597090507e-05 trainer/global_step‚7549W–Ì¥Æ & lr-AdamW/pg2‚3.754469597090507e-05 trainer/global_step‚7549 % _timestamp‚1.7884130378963542e+09 _runtime‚5648.308572545 _step‚618 & lr-AdamW/pg1‚3.754469597090507e-05ê”úcÔ2¦- _wandbruntime‚5648  _runtime‚5648VôÜé'§ãO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚213027389190144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99633351 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 101088290 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚7549 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58161556 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 101088792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10651710253056 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚213268681765844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚202647673846784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99633351 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10651879252992 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99633353 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42926736 _step‚619 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚502 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚5648.309465968 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42927236 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226136064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚236550247842304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10836151866880 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚223679099443200 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚223679099443200 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99633353 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152718619 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10621007919060 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 152718619 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚213027407193088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚213027407193088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚213027389190144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚236550473978368 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10651710253056 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚500 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚213268863926608 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10836354062848 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ƒöÒ1î'¨èO [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10621007919060 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 152718619 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚213027407193088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚213027407193088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚213027389190144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚236550473978368 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10651710253056 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚500 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚213268863926608 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10836354062848 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚213027389190144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99633351 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 101088290 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚7549 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58161556 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 101088792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10651710253056 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚213268681765844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚202647673846784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99633351 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10651879252992 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99633353 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42926736 _step‚619 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚502 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚5648.309465968 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42927236 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226136064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚236550247842304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10836151866880 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚223679099443200 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚223679099443200 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99633353 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152718619 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128ëqVîo2©- _wandbruntime‚5648  _runtime‚5648—»‰IªíM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42932167 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10837780782080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requesc³ÜÖª#ted_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _runtime‚5648.961426568 _step‚620 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 252364378 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10653103391232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚236580572843008 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚223707948962304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 152738432 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10622397083604 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚213054676557824 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10653272404480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 252364378 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10837563919360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252384444 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99646010 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚213296196273696 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99646010 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚213054658554880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58168933 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10653103391232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10622562205216 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚213054676557824 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 trainer/global_step‚7549 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99646012 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚225743008923648 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚202673617029120 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101101098 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚213296014112724 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 152718368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚223707761946112 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚213054658554880 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 152738432 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 101101588 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42932655 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚225743032863744 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚202673634068480 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972{¶Âø&«òM O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚213054676557824 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 trainer/global_step‚7549 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99646012 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚225743008923648 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚202673617029120 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101101098 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚213296014112724 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 152718368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚223707761946112 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚213054658554880 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 152738432 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 101101588 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42932655 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚225743032863744 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚202673634068480 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42932167 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10837780782080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _runtime‚5648.961426568 _step‚620 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 252364378 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10653103391232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚236580572843008 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚223707948962304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 152738432 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10622397083604 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚213054676557824 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10653272404480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 252364378 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10837563919360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252384444 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99646010 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚213296196273696 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99646010 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚213054658554880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58168933 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10653103391232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10622562205216ìíÏE°2¬- _wandbruntime‚5648  _runtime‚5648Ç êW­Ñ /loss/FRt2/assignment_loss‚2.614457230676304 0loss/FRt4/assignment_loss‚3.9227765847225577 /loss/FBt2/assignment_loss‚0.347473897936834 1loss/SRqqt1/detection_loss‚2.4565331505404577 _runtime‚ 5648.96287683 _step‚621 1loss/FBt3/assignment_loss‚0.19983447981735197 /loss/FRt1/detection_loss‚2.8984488292976662 /loss/FRt3/detection_loss‚1.9097006423719998 % _timestamp‚1.7884130385508263e+09 0loss/FBt1/assignment_loss‚0.2544177164910996 1loss/SRqqt3/detection_loss‚1.1614396827200713 /loss/FBt4/detection_loss‚2.3165060667430653 /loss/FRt3/assignment_loss‚2.694127369543602 1loss/SRqqt2/assignment_loss‚1.123420078593951 0loss/SRqqt4/detection_loss‚1.760256334800612 /loss/FBt3/detection_loss‚0.7523461552103982 2loss/SRqqt3/assignment_loss‚1.1391470790288205 /loss/FRt2/detection_loss‚2.2228713147342205 /loss/FBt1/detection_loss‚2.9822516820647498 trainer/global_step‚7549 .loss/FRt4/detection_loss‚2.249191421635297 1loss/SRqqt2/detection_loss‚1.5873696001676412 /loss/FBt2/detection_loss‚2.8076286017894745 %loss/total_loss‚46.60861642780897 2loss/SRqqt4/assignment_loss‚1.8319598051973365 0loss/FBt4/assignment_loss‚0.4045009851169037 epoch‚6 /loss/FRt1/assignment_loss‚4.611998920087461 1loss/SRqqt1/assignment_loss‚2.359958798521095›ø²—\®Ö /loss/FBt3/detection_loss‚0.7523461552103982 2loss/SRqqt3/assignment_loss‚1.1391470790288205 /loss/FRt2/detection_loss‚2.2228713147342205 /loss/FBt1/detection_loss‚2.9822516820647498 trainer/global_step‚7549 .loss/FRt4/detection_loss‚2.249191421635297 1loss/SRqqt2/detection_loss‚1.5873696001676412 /loss/FBt2/detection_loss‚2.8076286017894745 %loss/total_loss‚46.60861642780897 2loss/SRqqt4/assignment_loss‚1.8319598051973365 0loss/FBt4/assignment_loss‚0.4045009851169037 epoch‚6 /loss/FRt1/assignment_loss‚4.611998920087461 1loss/SRqqt1/assignment_loss‚2.359958798521095 /loss/FRt2/assignment_loss‚2.614457230676304 0loss/FRt4/assignment_loss‚3.9227765847225577 /loss/FBt2/assignment_loss‚0.347473897936834 1loss/SRqqt1/detection_loss‚2.4565331505404577 _runtime‚ 5648.96287683 _step‚621 1loss/FBt3/assignment_loss‚0.19983447981735197 /loss/FRt1/detection_loss‚2.8984488292976662 /loss/FRt3/detection_loss‚1.9097006423719998 % _timestamp‚1.7884130385508263e+09 0loss/FBt1/assignment_loss‚0.2544177164910996 1loss/SRqqt3/detection_loss‚1.1614396827200713 /loss/FBt4/detection_loss‚2.3165060667430653 /loss/FRt3/assignment_loss‚2.694127369543602 1loss/SRqqt2/assignment_loss‚1.123420078593951 0loss/SRqqt4/detection_loss‚1.760256334800612í©¥Ïœ¯‚(:‘ ôˆäÔܹ£™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚103.329' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.665799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sl °°‚(: ôˆäԦߋœ disk.rbd3.in‚0* proc.memory.percent‚17.569398880004883 disk.sda.out‚ 227.59375 disk.dm-1.in‚0 proc.cpu.threads‚17 network.sent‚ 1.7799643e+07 network.recv‚ 2.077056e+06 disk.sda3.out‚ 227.59375 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 710.40625 disk.dm-0.in‚ 710.40625" disk.dm-0.out‚66580.5322265625% proc.memory.availableMB‚ 251.53125 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 227.59375$ memory_percent‚98.46477508544922) disk./.usagePercent‚73.68935995617807' disk.nvme0c0n1.out‚66580.5322265625 disk.sda2.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 3.12890625 disk.rbd0.in‚0$ disk./.usageGB‚1098.024715423584 disk.sda.in‚0 disk.drbd1777.in‚0% disk.nvme0n1.out‚66580.5322265625 disk.nvme0n1.in‚ 710.40625 disk.rbd2.in‚0! disk.rbd2.out‚130.80810546875 disk.rbd3.out‚0" proc.memory.rssMB‚ 2878.5703125 cpu‚25.384463282117842 disk.rbd0.out‚0‡ÙÆ1œ±‚(:‘ ƒ‰äÔàÝ™ gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.775' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.755000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ôܻ沂(:” ƒ‰äÔÁ¾³› disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.12890625 disk.rbd3.out‚0' proc.memory.availableMB‚ 251.2890625 network.recv‚ 2.082636e+06 disk.dm-1.out‚ 228.2109375 disk.sda2.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 710.40625$ memory_percent‚98.46625328063965% disk./.usageGB‚1098.0247230529785" disk.dm-0.out‚66679.6103515625 disk.rbd0.out‚0 disk.rbd3.in‚0' disk.nvme0c0n1.out‚66679.6103515625 disk.nvme0n1.in‚ 710.40625% disk.nvme0n1.out‚66679.6103515625 disk.nvme0c0n1.in‚ 710.40625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 130.89453125 disk.sda.in‚0" proc.memory.rssMB‚ 2878.5703125 proc.cpu.threads‚17 network.sent‚ 1.7907786e+07) disk./.usagePercent‚73.68936046819313 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda1.in‚0* proc.memory.percent‚17.569398880004883 cpu‚25.38358870398466 disk.drbd1777.out‚0 disk.sda3.out‚ 228.2109375 disk.sda.out‚ 228.2109375”´ ¿2³- _wandbruntime‚5682  _runtime‚5682ÕÊ—ìÇ´Á _runtime‚5682.759348935 _step‚622 & lr-AdamW/pg1‚3.684125207023058e-05 & lr-AdamW/pg2‚3.684125207023058e-05 trainer/global_step‚7599 % _timestamp‚1.7884130723472223e+09Åe¶ÌµÆ _step‚622 & lr-AdamW/pg1‚3.684125207023058e-05 & lr-AdamW/pg2‚3.684125207023058e-05 trainer/global_step‚7599 % _timestamp‚1.7884130723472223e+09 _runtime‚5682.759348935î-¢`2¶- _wandbruntime‚5682  _runtime‚5682ùuX¯·çO % _timestamp‚1.7884130723479326e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚225112239002624 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10721381575680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚225112426005504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 trainer/global_step‚7599 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚214390875429888 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10721550575616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 R>DeviceStats a a> Monitor.on_train_batch_start/active_bytes.all.freed‚225112239002624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10690479737468 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153711321 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚227159743386624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100266303 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43197322 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 _step‚623 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 253957584 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 253977624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚227159767326720 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10907027136000 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153691283 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚203944850002944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416iF¼ˆò'¸ìO [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10721381575680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚225112426005504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 trainer/global_step‚7599 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚214390875429888 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10721550575616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚225112239002624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10690479737468 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153711321 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚227159743386624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100266303 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43197322 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 _step‚623 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 253957584 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 253977624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚227159767326720 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10907027136000 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153691283 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚203944850002944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 % _timestamp‚1.7884130723479326e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚225112239002624ï=ŠOÉ2¹- _wandbruntime‚5683  _runtime‚5683ê¹<¶÷&ºñM PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100278962 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100278960 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 253990030 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10722774713856 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254010096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10722943727104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚511 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚225141088521728 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43203232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58537781 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10722943727104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚214418144794624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 253990030 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚214662645047932 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚214418144794624 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153731134 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153711070 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚238097105647104 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10692034023624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153731134 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101740502 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153711070 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚214418126791680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚214662827208904 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚225140901505536 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚513 % _timestamp‚1.7884130729924405e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10722774713856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100278960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚203970776145920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58537783çPŠ„%»öM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 254010096 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100278962 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚238097338061312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153731134 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101740502 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153711070 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚214418126791680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚214662827208904 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚225140901505536 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚513 % _timestamp‚1.7884130729924405e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10722774713856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100278960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚203970776145920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58537783 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100278962 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100278960 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 253990030 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10722774713856 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254010096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10722943727104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚511 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚225141088521728 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43203232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58537781 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10722943727104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚214418144794624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 253990030 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚214662645047932 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚214418144794624 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153731134 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153711070 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚238097105647104 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10692034023624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚226545378562048 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10791221898240 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154704023 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚228605414737920 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10759951555876 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100899253 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10791052898304 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚239583341220864 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚226545565564928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 154704023 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100899253 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102366358 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10977950423040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚239583579939840 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10978165201920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58899254 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 255583236 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚215754343666688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚215754325663744 _runtime‚5717.744278736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚226545565564928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚205241992080384 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚215754325663744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚228605390797824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58899256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43467102 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚205242009119744 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚216002125797024 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚216001943636260 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10760116677280 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0u¾ò'ÊìO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚226545378562048 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10791221898240 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154704023 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚228605414737920 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10759951555876 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100899253 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10791052898304 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚239583341220864 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚226545565564928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 154704023 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100899253 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102366358 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10977950423040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚239583579939840 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10978165201920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58899254 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 255583236 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚215754343666688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚215754325663744 _runtime‚5717.744278736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚226545565564928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚205241992080384 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚215754325663744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚228605390797824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58899256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43467102 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚205242009119744 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚216002125797024 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚216001943636260 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10760116677280 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43466606 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7884131073318174e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154683985 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚215781613031424 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚215781613031424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58906631 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10979592234496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚228634303746048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚216029275983140 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43471791 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚628 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚226574041064960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100911912 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 % _timestamp‚1.7884131079938893e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10761340720420 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 255635748 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚226574228081152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215781595028480 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚521 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 255615682 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154723836 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102378943 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102378422 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43472310 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚228634327686144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _runtime‚5718.406539924 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255635748 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚226574228081152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58906633 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100911910 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154723836 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚239613679117824 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0›U0ü&ÍöM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚205267935262720 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10761505842032 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 % _timestamp‚1.7884131079938893e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10761340720420 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 255635748 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚226574228081152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215781595028480 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚521 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 255615682 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154723836 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102378943 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102378422 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43472310 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚228634327686144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _runtime‚5718.406539924 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255635748 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚226574228081152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58906633 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100911910 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154723836 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚239613679117824 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚216029458144112 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚215781613031424 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚215781613031424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58906631 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10979592234496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚228634303746048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚216029275983140 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43471791 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚628 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚226574041064960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100911912 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 5752.33962827 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10860893220864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59268104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103005704 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚7699 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257228928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚217117811903488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 257208888 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155696725 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155696725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43737598 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚217117793900544 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10860724220928 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227978518121472 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10860893220864 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10829423374284 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚241100133245440uy‰ÚêO JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10860893220864 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10829423374284 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚241100133245440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚241099900817920 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11048862608896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚217368756732232 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚206539151197184 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚217117811903488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 5752.33962827 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10860893220864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59268104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103005704 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚7699 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257228928 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚217117811903488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 257208888 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155696725 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155696725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43737598 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚217117793900544 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10860724220928 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227978518121472÷?éyâ2Û- _wandbruntime‚5753  _runtime‚5753ÐN”Åö&ÜðM M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884131425909925e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 257241334 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10862117359104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚228007367640576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10862117359104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚206565111418880 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10830812538828 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚228007180624384 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155696474 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103018368 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103017823 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155716538 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43742342 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚241130467803136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 257241334 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101544860 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43742885 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257261400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11050492705792 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155696474 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101544862 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚217145081268224 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0xüÂÛû&ÝõM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚217145081268224 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚230079975097344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10862286372352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10862286372352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚230079951157248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚217396089079320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚206565111418880 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10830812538828 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚228007180624384 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155696474 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103018368 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103017823 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155716538 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43742342 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚241130467803136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 257241334 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101544860 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43742885 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257261400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11050492705792 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155696474 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101544862 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚217145081268224 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884131425909925e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 257241334 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10862117359104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚228007367640576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10862117359104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚218735387667440 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 103644715 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11119717431808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚229411657680896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102165151 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59636954 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 $ _timestamp‚1.788413176954131e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚218481262137344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚229411657680896 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59636956 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11119928016384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44007761 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚231496685620224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚218735205506676 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚207836310313984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚635 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103645248 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103645248 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚218735387667440 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 103644715 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11119717431808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚229411657680896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102165151 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59636954 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 $ _timestamp‚1.788413176954131e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚218481262137344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚229411657680896 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59636956 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11119928016384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44007761 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚231496685620224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚218735205506676 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚207836310313984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚635 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103657492 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10900449478848 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚493 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚218762720014528 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7884131776090484e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59644331 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚231525622508544 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156709240 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚491 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚207862270535680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 156689176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10931957694976 ZFDeviceStatsMon-Î'{itor.on_train_batch_end/allocated_bytes.large_pool.freed‚218508531502080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚636 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚229440507200000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚231525598568448 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10931788681728 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258887052 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156709240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚207862253496320 fµ{ü&íöM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156709240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚207862253496320 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚229440507200000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10900284357236 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚218508549505024 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10931788681728 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102177812 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚242646970283008 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚242646737868800 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚218762537853556 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11121139300352 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102177812 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102177810 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 258866986 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103657492 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10900449478848 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚493 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚218762720014528 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7884131776090484e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59644331 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚231525622508544 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156709240 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚491 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚207862270535680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 156689176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10931957694976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚218508531502080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚636 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚229440507200000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚231525598568448 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10931788681728 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258887052ü‹ ¼÷2î- _wandbruntime‚5788  _runtime‚5788É‹!XïÒ /loss/FRt2/assignment_loss‚2.810468513591617 0loss/SRqqt2/assignment_loss‚1.34960491628601 0loss/SRqqt4/detection_loss‚1.909027709605846 %loss/total_loss‚47.40268061643272 1loss/SRqqt3/assignment_loss‚1.042778866016306 /loss/FRt1/detection_loss‚3.0703535310683714 % _timestamp‚1.7884131776182165e+09 2loss/SRqqt4/assignment_loss‚1.5592967741032864 0loss/FBt1/assignment_loss‚0.4085473602928687 /loss/FBt3/detection_loss‚0.7758483513593092 _step‚637 0loss/FRt4/assignment_loss‚3.3520813294819423 /loss/FRt3/detection_loss‚2.0759386694431305 1loss/SRqqt2/detection_loss‚1.5666393079436742 /loss/FBt4/detection_loss‚2.3770910887157215 /loss/FRt1/assignment_loss‚4.496531132728823 /loss/FRt3/assignment_loss‚2.092458184957504 1loss/FBt3/assignment_loss‚0.31360731954702603 /loss/FRt2/detection_loss‚2.2521368452146944 /loss/FRt4/detection_loss‚2.3754381809915817 0loss/SRqqt3/detection_loss‚1.276548206107691 epoch‚6 _runtime‚5788.030288985 1loss/FBt2/assignment_loss‚0.37828112712928225 0loss/FBt4/assignment_loss‚0.4975750859578993 1loss/SRqqt1/detection_loss‚2.7024774956703186 .loss/FBt1/detection_loss‚2.749024232228597 2loss/SRqqt1/assignment_loss‚2.3990453577041624 /loss/FBt2/detection_loss‚3.5718810302870616 trainer/global_step‚7749B©×£]ð× 1loss/SRqqt3/assignment_loss‚1.042778866016306 /loss/FRt1/detection_loss‚3.0703535310683714 % _timestamp‚1.7884131776182165e+09 2loss/SRqqt4/assignment_loss‚1.5592967741032864 0loss/FBt1/assignment_loss‚0.4085473602928687 /loss/FBt3/detection_loss‚0.7758483513593092 _step‚637 0loss/FRt4/assignment_loss‚3.3520813294819423 /loss/FRt3/detection_loss‚2.0759386694431305 1loss/SRqqt2/detection_loss‚1.5666393079436742 /loss/FBt4/detection_loss‚2.3770910887157215 /loss/FRt1/assignment_loss‚4.496531132728823 /loss/FRt3/assignment_loss‚2.092458184957504 1loss/FBt3/assignment_loss‚0.31360731954702603 /loss/FRt2/detection_loss‚2.2521368452146944 /loss/FRt4/detection_loss‚2.3754381809915817 0loss/SRqqt3/detection_loss‚1.276548206107691 epoch‚6 _runtime‚5788.030288985 1loss/FBt2/assignment_loss‚0.37828112712928225 0loss/FBt4/assignment_loss‚0.4975750859578993 1loss/SRqqt1/detection_loss‚2.7024774956703186 .loss/FBt1/detection_loss‚2.749024232228597 2loss/SRqqt1/assignment_loss‚2.3990453577041624 /loss/FBt2/detection_loss‚3.5718810302870616 trainer/global_step‚7749 /loss/FRt2/assignment_loss‚2.810468513591617 0loss/SRqqt2/assignment_loss‚1.34960491628601 0loss/SRqqt4/detection_loss‚1.909027709605846 %loss/total_loss‚47.40268061643272ýK‚Œ6œñ‚(:‘ û‰äÔ¬­º™ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.969' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.993799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üöÒò‚(: û‰äÔ¹´œ disk.rbd3.out‚0( disk.nvme0c0n1.out‚67280.70458984375 disk.sda1.in‚0 proc.cpu.threads‚17! disk.rbd2.out‚134.56298828125& disk.nvme0n1.out‚67280.70458984375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚711.0625 disk.sda2.in‚0 disk.rbd1.in‚0$ memory_percent‚97.97210693359375 cpu‚25.37665204222876) disk./.usagePercent‚73.68957756258216 disk.nvme0n1.in‚711.0625 disk.rbd3.in‚0 disk.sda.out‚ 232.640625 disk.drbd1777.out‚0 network.sent‚ 1.8261619e+07 disk.dm-1.in‚0 disk.dm-1.out‚ 232.640625 disk.sda3.in‚0" proc.memory.rssMB‚ 2797.8671875* proc.memory.percent‚17.076826095581055 disk.dm-0.in‚711.0625 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0" proc.memory.availableMB‚332.25# disk.dm-0.out‚67280.70458984375 disk.rbd2.in‚0 disk.sda3.out‚ 232.640625 disk.sda.in‚0 disk.drbd1777.in‚0 disk.rbd1.out‚ 3.12890625 network.recv‚ 2.121372e+06% disk./.usageGB‚1098.0279579162598\"Ó³‘ó‚(:† ŠŠäÔÚ˜¸™ gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.423' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0846 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dËæD#ô‚(:˜ ŠŠäÔ¤ðá› disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 network.sent‚ 1.8370348e+07% disk./.usageGB‚1098.0280418395996( disk.nvme0c0n1.out‚67380.05810546875 disk.sda3.in‚0 disk.sda.out‚ 233.0703125 disk.drbd1777.out‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 331.79296875) disk./.usagePercent‚73.68958319474793 disk.sda3.out‚ 233.0703125 disk.nvme0n1.in‚711.0625# disk.dm-0.out‚67380.05810546875! disk.rbd2.out‚134.57861328125$ memory_percent‚97.97489643096924* proc.memory.percent‚17.076826095581055 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.rbd1.out‚ 3.12890625 disk.dm-0.in‚711.0625 disk.rbd3.in‚0 disk.dm-1.out‚ 233.0703125 cpu‚25.375817399965193 network.recv‚ 2.12804e+06 disk.sda2.out‚0& disk.nvme0n1.out‚67380.05810546875 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚711.0625" proc.memory.rssMB‚ 2797.8671875àòHõ‚(:… ™ŠäÔÿûª™ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.235' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.647 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0͇’'ö‚(:œ ™ŠäÔ̼✠disk.sda.out‚ 233.78515625 disk.rbd1.out‚ 3.12890625# disk.dm-0.out‚67479.21630859375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚711.0625( proc.memory.availableMB‚ 331.79296875) disk./.usagePercent‚73.68958191471025$ memory_percent‚97.97489643096924 proc.cpu.threads‚17% disk./.usageGB‚1098.0280227661133 disk.rbd1.in‚0& disk.nvme0n1.out‚67479.21630859375 disk.rbd0.out‚0 disk.rbd3.in‚0" proc.memory.rssMB‚ 2797.8671875 network.sent‚ 1.8376486e+07 disk.dm-1.out‚ 233.78515625 disk.rbd2.in‚0! disk.rbd2.out‚134.71142578125 disk.rbd3.out‚0 cpu‚25.374929365852328 network.recv‚ 2.132493e+06( disk.nvme0c0n1.out‚67479.21630859375 disk.sda2.in‚0 disk.sda3.out‚ 233.78515625 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚711.0625 disk.sda3.in‚0* proc.memory.percent‚17.076826095581055 disk.drbd1777.out‚0 disk.dm-0.in‚711.0625 disk.dm-1.in‚0 disk.sda2.out‚0'Èc2÷- _wandbruntime‚5822  _runtime‚5822‰ìåÇøÁ trainer/global_step‚7799 % _timestamp‚1.7884132116595333e+09 _runtime‚5822.071667028 _step‚638 & lr-AdamW/pg1‚3.405743420371351e-05 & lr-AdamW/pg2‚3.405743420371351e-05¼ßK—ÌùÆ & lr-AdamW/pg1‚3.405743420371351e-05 & lr-AdamW/pg2‚3.405743420371351e-05 trainer/global_step‚7799 % _timestamp‚1.7884132116595333e+09 _runtime‚5822.071667028 _step‚638þ¥í'ûçO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10968532132504 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚219844730374144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102798103 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 157682129 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157682129 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230844797240320 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚220102018602648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚220101836441884 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚209133486470144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11190720955392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 260480232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚513 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚219844748377088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157662091 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104284864 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44279573 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚230844797240320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11000066866176 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚230844984243200 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚7799 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11000235866112 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 260460192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚515 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232942356971520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102798103 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102798101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244133077926912 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230844797240320 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚220102018602648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚220101836441884 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚209133486470144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11190720955392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 260480232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚513 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚219844748377088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157662091 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104284864 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44279573 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚230844797240320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11000066866176 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚230844984243200 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚7799ÿ^ýE~2ý- _wandbruntime‚5822  _runtime‚5822B&z÷&þñM M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚209159429652480 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11001460004352 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚244163400671744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 260492638 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚219871999738880 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11001460004352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104298215 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157681878 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157701942 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260512704 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 _step‚640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚230873646759424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44284516 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 trainer/global_step‚7799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11192130752000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚230873646759424 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260492638 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚5822.930618479 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10969756175644 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260512704 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104297697 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚516 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚232971245979648 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11191920180736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚219871999738880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11001629017600 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚00¶ôÓü&ÿöM F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚232971245979648 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11191920180736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚219871999738880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11001629017600 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚209159429652480 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11001460004352 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚244163400671744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 260492638 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚219871999738880 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11001460004352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104298215 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157681878 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157701942 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260512704 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 _step‚640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚230873646759424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44284516 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 trainer/global_step‚7799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11192130752000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚230873646759424 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260492638 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚5822.930618479 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10969756175644 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260512704 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104297697 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚516 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972€ñüe@2€- _wandbruntime‚5822  _runtime‚5822Béê4YÓ 2loss/SRqqt3/assignment_loss‚1.1455966355279088 /loss/FBt4/assignment_loss‚0.508739500961123 .loss/FRt2/detection_loss‚2.274213126897812 0loss/FRt4/assignment_loss‚3.3900976673914838 2loss/SRqqt4/assignment_loss‚1.4850070335123005 0loss/SRqqt1/detection_loss‚2.495387100404309 /loss/FBt1/detection_loss‚2.9412294951352207 _runtime‚5822.931698353 0loss/FBt2/assignment_loss‚0.3107180698602211 1loss/SRqqt4/detection_loss‚1.8970396543035701 1loss/FBt3/assignment_loss‚0.24576502072503673 /loss/FBt4/detection_loss‚2.1632881930896213 0loss/FRt2/assignment_loss‚3.0958185565471648 epoch‚6 % _timestamp‚1.7884132125196743e+09 /loss/FRt1/assignment_loss‚4.494963884353638 0loss/FRt3/assignment_loss‚1.6793513777000564 0loss/FBt1/assignment_loss‚0.1372428825445902 .loss/FBt2/detection_loss‚3.512475941862379 %loss/total_loss‚45.96679422423053 trainer/global_step‚7799 .loss/FRt1/detection_loss‚3.155362555980682 /loss/FRt3/detection_loss‚1.6913731079881213 1loss/SRqqt2/detection_loss‚1.6530400322841816 1loss/SRqqt3/detection_loss‚1.4376640563520293 _step‚641 2loss/SRqqt1/assignment_loss‚2.3950155896525227 2loss/SRqqt2/assignment_loss‚1.1338882227046567 .loss/FRt4/detection_loss‚2.055882565390605 /loss/FBt3/detection_loss‚0.6676339530613025Žƒ˜b^‚Ø 0loss/FBt2/assignment_loss‚0.3107180698602211 1loss/SRqqt4/detection_loss‚1.8970396543035701 1loss/FBt3/assignment_loss‚0.24576502072503673 /loss/FBt4/detection_loss‚2.1632881930896213 0loss/FRt2/assignment_loss‚3.0958185565471648 epoch‚6 % _timestamp‚1.7884132125196743e+09 /loss/FRt1/assignment_loss‚4.494963884353638 0loss/FRt3/assignment_loss‚1.6793513777000564 0loss/FBt1/assignment_loss‚0.1372428825445902 .loss/FBt2/detection_loss‚3.512475941862379 %loss/total_loss‚45.96679422423053 trainer/global_step‚7799 .loss/FRt1/detection_loss‚3.155362555980682 /loss/FRt3/detection_loss‚1.6913731079881213 1loss/SRqqt2/detection_loss‚1.6530400322841816 1loss/SRqqt3/detection_loss‚1.4376640563520293 _step‚641 2loss/SRqqt1/assignment_loss‚2.3950155896525227 2loss/SRqqt2/assignment_loss‚1.1338882227046567 .loss/FRt4/detection_loss‚2.055882565390605 /loss/FBt3/detection_loss‚0.6676339530613025 2loss/SRqqt3/assignment_loss‚1.1455966355279088 /loss/FBt4/assignment_loss‚0.508739500961123 .loss/FRt2/detection_loss‚2.274213126897812 0loss/FRt4/assignment_loss‚3.3900976673914838 2loss/SRqqt4/assignment_loss‚1.4850070335123005 0loss/SRqqt1/detection_loss‚2.495387100404309 /loss/FBt1/detection_loss‚2.9412294951352207 _runtime‚5822.931698353úW꜃‚(:‘ ¨ŠäÔÓ”¬™ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚120.138' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.027600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$ùá$„‚(:™ ¨ŠäÔÎÌÞœ disk.rbd3.in‚0 disk.sda3.out‚ 234.12109375 disk.sda.in‚0 disk.drbd1777.in‚0 disk.rbd1.out‚ 3.12890625 network.sent‚ 1.8382241e+07 disk.nvme0c0n1.in‚711.0625 disk.sda2.in‚0( proc.memory.availableMB‚ 332.00390625 cpu‚25.374097895992843) disk./.usagePercent‚73.68958191471025% disk./.usageGB‚1098.0280227661133% disk.nvme0n1.out‚67479.9326171875 disk.rbd3.out‚0" proc.memory.rssMB‚ 2797.8671875 disk.nvme0n1.in‚711.0625 disk.dm-1.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda3.in‚0 disk.dm-1.out‚ 234.12109375! disk.rbd2.out‚134.72705078125 disk.sda.out‚ 234.12109375 disk.sda1.out‚0 disk.drbd1777.out‚0$ memory_percent‚97.97360897064209* proc.memory.percent‚17.076826095581055 disk.dm-0.in‚711.0625 disk.rbd0.in‚0 disk.rbd2.in‚0' disk.nvme0c0n1.out‚67479.9326171875 network.recv‚ 2.136557e+06" disk.dm-0.out‚67479.93261718750,Ž­›…‚(: ·ŠäÔØ·ž™ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚92.144' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.428800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉuI †‚(:• ·ŠäÔ¶å‚ disk.rbd3.in‚0' disk.nvme0c0n1.out‚67578.9521484375$ memory_percent‚97.97213077545166( proc.memory.availableMB‚ 332.24609375 network.recv‚ 2.142129e+06" disk.dm-0.out‚67578.9521484375* proc.memory.percent‚17.076826095581055 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 234.6328125 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚134.8291015625 disk.nvme0c0n1.in‚711.0625 cpu‚25.373310320359806 network.sent‚ 1.8490313e+07) disk./.usagePercent‚73.68958191471025% disk./.usageGB‚1098.0280227661133 disk.sda.out‚ 234.6328125% disk.nvme0n1.out‚67578.9521484375 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0" proc.memory.rssMB‚ 2797.8671875 proc.cpu.threads‚17 disk.rbd1.out‚ 3.12890625 disk.dm-0.in‚711.0625 disk.dm-1.out‚ 234.6328125 disk.sda3.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.nvme0n1.in‚711.0625Õ)52‡- _wandbruntime‚5856  _runtime‚5856âäóLjÁ _step‚642 & lr-AdamW/pg1‚3.336969436558228e-05 & lr-AdamW/pg2‚3.336969436558228e-05 trainer/global_step‚7849 % _timestamp‚1.7884132465532658e+09 _runtime‚5856.965405827`K¤?P‰Æ & lr-AdamW/pg1‚3.336969436558228e-05 & lr-AdamW/pg2‚3.3369694365582!²(@|28e-05 trainer/global_step‚7849 % _timestamp‚1.7884132465532658e+09 _runtime‚5856.965405827 _step‚642‚WöeK2Š- _wandbruntime‚5856  _runtime‚5856hΓí'‹çO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103431051 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚232277936799744 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚232277936799744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44549848 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60374656 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚245649550220288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11037838829508 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 262085844 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158674831 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 trainer/global_step‚7849 _runtime‚5856.966332072 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158654793 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11069738188800 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚232278123802624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚234388004382720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚221468649537856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚210430628547584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11261545837568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚221468467377092 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103431051 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 104925021 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚245649309404160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚234387980442624 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103431053 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚519 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262105884 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104924502 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚221208216613888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚221208216613888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11069907188736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _step‚643 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚221208198610944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚221208198610944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60374654 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚517 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104924502 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚221208216613888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚221208216613888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11069907188736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _step‚643 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚221208198610944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚221208198610944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60374654 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚517 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚232277936799744 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚232277936799744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44549848 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60374656 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚245649550220288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11037838829508 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 262085844 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158674831 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 trainer/global_step‚7849 _runtime‚5856.966332072 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158654793 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11069738188800 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚232278123802624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚234388004382720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚221468649537856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚210430628547584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11261545837568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚221468467377092 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103431051 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 104925021 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚245649309404160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚234387980442624 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103431053 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚519 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262105884ƒ|DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104937282 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11071300340224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158694644 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11071131326976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚234416893390848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44555251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚221235485978624 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103443712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚210456571729920 _runtime‚5857.609836527 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚644 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚232306786318848 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158674580 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103443710 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚221495799723972 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11039227994052 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158674580 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103443710 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262138356 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚221235467975680 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104937764 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60382031 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11262746198528 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 $ _timestamp‚1.788413247197194e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚480 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264»Q¦õM OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158674580 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103443710 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262138356 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚221235467975680 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104937764 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 RSRsœáEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60382031 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11262746198528 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 $ _timestamp‚1.788413247197194e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚480 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚232306599302656 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104937282 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11071300340224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158694644 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11071131326976 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚234416893390848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44555251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚221235485978624 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103443712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚210456571729920 _runtime‚5857.609836527 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚644 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚232306786318848 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158674580 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103443710 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚221495799723972 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11039227994052 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460„ƒœ¥2- _wandbruntime‚5857  _runtime‚5857ŸiÍôY‘Ó 2loss/SRqqt4/assignment_loss‚1.6622218278547127 /loss/FRt4/detection_loss‚2.7703436596973523 /loss/FRt4/assignment_loss‚4.161413318402058 0loss/FBt1/assignment_loss‚0.5697346078165667 1loss/FBt4/assignment_loss‚0.39566416785764885 /loss/FBt4/detection_loss‚2.6744070768356325 0loss/FRt2/assignment_loss‚2.8400896108427713 1loss/SRqqt2/detection_loss‚1.5395292027250809 0loss/SRqqt3/detection_loss‚1.634597051423043 trainer/global_step‚7849 _step‚645 0loss/SRqqt4/detection_loss‚2.153960881488664 /loss/FBt2/detection_loss‚2.8546875827014446 epoch‚6 % _timestamp‚1.7884132471990688e+09 2loss/SRqqt2/assignment_loss‚1.2239949771148317 0loss/FBt2/assignment_loss‚0.5080572278027944 1loss/FBt3/assignment_loss‚0.26057258364956887 .loss/FRt1/detection_loss‚2.853149550035596 /loss/FRt3/detection_loss‚1.9620087907022359 /loss/FBt3/detection_loss‚0.7574485006504147 .loss/FRt1/assignment_loss‚5.35340242087841 0loss/FRt3/assignment_loss‚1.9632349963090858 2loss/SRqqt3/assignment_loss‚1.1035720484796911 /loss/FRt2/detection_loss‚2.4489605610453804 0loss/SRqqt1/detection_loss‚2.734079822897911 /loss/FBt1/detection_loss‚2.6626382299831937 %loss/total_loss‚49.63409030638462 _runtime‚5857.611143089 1loss/SRqqt1/assignment_loss‚2.546321609190532Qö^’Ø 0loss/FRt2/assignment_loss‚2.8400896108427713 1loss/SRqqt2/detection_loss‚1.5395292027250809 0loss/SRqqt3/detection_loss‚1.634597051423043 trainer/global_step‚7849 _step‚645 0loss/SRqqt4/detection_loss‚2.153960881488664 /loss/FBt2/detection_loss‚2.8546875827014446 epoch‚6 % _timestamp‚1.7884132471990688e+09 2loss/SRqqt2/assignment_loss‚1.2239949771148317 0loss/FBt2/assignment_loss‚0.5080572278027944 1loss/FBt3/assignment_loss‚0.26057258364956887 .loss/FRt1/detection_loss‚2.853149550035596 /loss/FRt3/detection_loss‚1.9620087907022359 /loss/FBt3/detection_loss‚0.7574485006504147 .loss/FRt1/assignment_loss‚5.35340242087841 0loss/FRt3/assignment_loss‚1.9632349963090858 2loss/SRqqt3/assignment_loss‚1.1035720484796911 /loss/FRt2/detection_loss‚2.4489605610453804 0loss/SRqqt1/detection_loss‚2.734079822897911 /loss/FBt1/detection_loss‚2.6626382299831937 %loss/total_loss‚49.63409030638462 _runtime‚5857.611143089 1loss/SRqqt1/assignment_loss‚2.546321609190532 2loss/SRqqt4/assignment_loss‚1.6622218278547127 /loss/FRt4/detection_loss‚2.7703436596973523 /loss/FRt4/assignment_loss‚4.161413318402058 0loss/FBt1/assignment_loss‚0.5697346078165667 1loss/FBt4/assignment_loss‚0.39566416785764885 /loss/FBt4/detection_loss‚2.6744070768356325…Û€ÎÜœ“‚(:‘ ÆŠäÔ÷Éš gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.772' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.754400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ĵµ”‚(:“ ÆŠäÔç÷› disk.sda1.in‚0 disk.drbd1777.out‚0! disk.dm-0.out‚67678.369140625 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.dm-1.in‚0 network.recv‚ 2.146927e+06 disk.sda.in‚0 disk.dm-0.in‚711.0625 disk.sda3.out‚ 235.05078125" proc.memory.rssMB‚ 2797.8671875* proc.memory.percent‚17.076826095581055 cpu‚25.372454216032356 proc.cpu.threads‚17 network.sent‚ 1.8496558e+07$ disk./.usageGB‚1097.855411529541 disk.rbd2.out‚134.8447265625 disk.rbd3.out‚0 disk.nvme0c0n1.in‚711.0625 disk.drbd1777.in‚0$ disk.nvme0n1.out‚67678.369140625 disk.dm-1.out‚ 235.05078125) disk./.usagePercent‚73.67799782979341 disk.sda.out‚ 235.05078125 disk.nvme0n1.in‚711.0625& disk.nvme0c0n1.out‚67678.369140625# memory_percent‚97.9752779006958( proc.memory.availableMB‚ 331.73046875 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.12890625 disk.rbd0.in‚0 disk.sda3.in‚0'ääf‘•‚(:† ÕŠäÔÈؘ™ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.872' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’¿Û–‚(:“ ÕŠäÔÁþœ disk.rbd3.in‚0 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚711.0625! disk.dm-0.out‚67678.685546875 disk.rbd2.out‚136.4814453125( proc.memory.availableMB‚ 331.44921875 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 235.66015625 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 3.12890625 disk.nvme0n1.in‚711.0625 disk.nvme0c0n1.in‚711.0625 disk.sda2.in‚0 disk.sda3.out‚ 235.66015625" proc.memory.rssMB‚ 2797.8671875 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚67678.685546875 disk.rbd2.in‚0$ memory_percent‚97.97699451446533 network.recv‚ 2.153241e+06$ disk./.usageGB‚1097.855728149414 disk.sda2.out‚0 disk.sda.out‚ 235.66015625 disk.sda1.in‚0 disk.drbd1777.out‚0* proc.memory.percent‚17.076826095581055 network.sent‚ 1.8605199e+07) disk./.usagePercent‚73.67801907841876& disk.nvme0c0n1.out‚67678.685546875 cpu‚25.37163429884195Y¯I2—- _wandbruntime‚5891  _runtime‚5891ó`ÌÌǘÁ _runtime‚5891.587311488 _step‚646 & lr-AdamW/pg1‚3.268552808101051e-05 & lr-AdamW/pg2‚3.268552808101051e-05 trainer/global_step‚7899 % _timestamp‚1.7884132811750815e+09 zR|Ì™Æ trainer/global_step‚7899 % _timestamp‚1.7884132811750815e+09 _runtime‚5891.587311488 _step‚646 & lr-AdamW/pg1‚3.268552808101051e-05 & lr-AdamW/pg2‚3.268552808101051e-05†Û[D72š- _wandbruntime‚5891  _runtime‚5891¯õˆtì'›æO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚247166059820544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚222835280473064 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 159667533 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 159647495 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚222571666847744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚503 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚222571666847744 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11332408026624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚7899 % _timestamp‚1.7884132811759171e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 159667533 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚233711263362048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚233711263362048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105564427 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60743506 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104064003 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44821426 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚222835098312300 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚211727804703744 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11139409511424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104064003 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚233711076359168 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 159647495 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11107475769320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚222571684850688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚222571684850688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11107310647916 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199èWÍUˆ œëO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104064003 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44821426 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚222835098312300 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚211727804703744 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11139409511424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104064003 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚233711076359168 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 159647495 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11107475769320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚222571684850688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚222571684850688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11107310647916 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚247166059820544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚222835280473064 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 159667533 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 159647495 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚222571666847744 XJDe–|iviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚503 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚222571666847744 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11332408026624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚7899 % _timestamp‚1.7884132811759171e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 159667533 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚233711263362048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚233711263362048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105564427 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60743506 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2‡P†Ž2- _wandbruntime‚5892  _runtime‚5892Û Mö&žðM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚233739925878272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚222598954215424 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚235862540802048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚211753747886080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11140802649600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788413281824735e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44826685 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228219904 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11333623339008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11108864934072 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11140971662848 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60750881 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚235862564742144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263764008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚511 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚648 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44826174 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11140971662848 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚247196392360960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚222598936212480 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚513 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚222862612820152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263743942 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104076660 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11333827618816 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚211753730846720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 trainer/global_step‚7899 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263743942 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105577568 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11108699812460 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263764008 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105577055 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚233739738862080 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11140802649600 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159667282 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60750883 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159667282©iĬû&ŸõM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚233739925878272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚222598954215424 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚235862540802048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚211753747886080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11140802649600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788413281824735e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44826685 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228219904 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11333623339008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11108864934072 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11140971662848 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60750881 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚235862564742144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263764008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚511 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚648 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44826174 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11140971662848 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚247196392360960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚222598936212480 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚513 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚222862612820152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263743942 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104076660 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11333827618816 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚211753730846720 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 trainer/global_step‚7899 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263743942 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105577568 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11108699812460 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263764008 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105577055 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚233739738862080 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11140802649600 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159667282 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60750883 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159667282 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0ˆs2 - _wandbruntime‚5892  _runtime‚5892Ö¥ïiY¡Ó .loss/FBt1/detection_loss‚3.275692045688629 /loss/FBt4/detection_loss‚2.8618786931037903 0loss/FRt3/assignment_loss‚2.2679697469223377 1loss/SRqqt1/detection_loss‚2.6688119847984875 trainer/global_step‚7899 2loss/SRqqt3/assignment_loss‚1.2078078595199147 .loss/FBt2/assignment_loss‚0.55208123821233 0loss/FBt3/assignment_loss‚0.2495357092108558 0loss/FBt4/assignment_loss‚0.9597519038245081 /loss/FBt3/detection_loss‚0.6086360612878505 %loss/total_loss‚48.47024206317046 /loss/FRt2/assignment_loss‚2.496306759970529 0loss/FRt4/assignment_loss‚3.8540308552759663 epoch‚6 % _timestamp‚1.7884132818263657e+09 _runtime‚5892.238397298 _step‚649 1loss/SRqqt4/detection_loss‚1.8271960184762353 1loss/FBt1/assignment_loss‚0.21943463432471616 .loss/FRt1/detection_loss‚3.317992150783539 /loss/FRt3/detection_loss‚1.9065471655407618 2loss/SRqqt2/assignment_loss‚1.4318274534855258 2loss/SRqqt4/assignment_loss‚1.6222055500144499 1loss/SRqqt2/detection_loss‚1.5847977943280165 1loss/SRqqt3/detection_loss‚1.5391452432591088 /loss/FBt2/detection_loss‚2.6978420530046736 /loss/FRt1/assignment_loss‚4.721344411373138 2loss/SRqqt1/assignment_loss‚2.3077417804914364 .loss/FRt2/detection_loss‚2.276720404624939 /loss/FRt4/detection_loss‚2.0149445456487163d\ˆ9^¢Ø 1loss/SRqqt1/detection_loss‚2.6688119847984875 trainer/global_step‚7899 2loss/SRqqt3/assignment_loss‚1.2078078595199147 .loss/FBt2/assignment_loss‚0.55208123821233 0loss/FBt3/assignment_loss‚0.2495357092108558 0loss/FBt4/assignment_loss‚0.9597519038245081 /loss/FBt3/detection_loss‚0.6086360612878505 %loss/total_loss‚48.47024206317046 /loss/FRt2/assignment_loss‚2.496306759970529 0loss/FRt4/assignment_loss‚3.8540308552759663 epoch‚6 % _timestamp‚1.7884132818263657e+09 _runtime‚5892.238397298 _step‚649 1loss/SRqqt4/detection_loss‚1.8271960184762353 1loss/FBt1/assignment_loss‚0.21943463432471616 .loss/FRt1/detection_loss‚3.317992150783539 /loss/FRt3/detection_loss‚1.9065471655407618 2loss/SRqqt2/assignment_loss‚1.4318274534855258 2loss/SRqqt4/assignment_loss‚1.6222055500144499 1loss/SRqqt2/detection_loss‚1.5847977943280165 1loss/SRqqt3/detection_loss‚1.5391452432591088 /loss/FBt2/detection_loss‚2.6978420530046736 /loss/FRt1/assignment_loss‚4.721344411373138 2loss/SRqqt1/assignment_loss‚2.3077417804914364 .loss/FRt2/detection_loss‚2.276720404624939 /loss/FRt4/detection_loss‚2.0149445456487163 .loss/FBt1/detection_loss‚3.275692045688629 /loss/FBt4/detection_loss‚2.8618786931037903 0loss/FRt3/assignment_loss‚2.2679697469223377‰Å#Èdœ£‚(:‘ äŠäÔ«ˆ¶› gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.564' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.912799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓÌùb¤‚(:‘ äŠäÔÏÆÉ network.sent‚ 1.8609646e+07% disk./.usageGB‚1097.8576698303223 disk.sda2.in‚0 disk.sda1.in‚0# disk.nvme0n1.out‚67778.05859375( proc.memory.availableMB‚ 332.26171875 disk.sda1.out‚0 disk.rbd1.out‚ 3.12890625 disk.rbd0.out‚0 disk.rbd2.in‚0) disk./.usagePercent‚73.67814938625368 disk.rbd3.in‚0% disk.nvme0c0n1.out‚67778.05859375 disk.sda3.out‚ 236.08984375 network.recv‚ 2.156488e+06 disk.nvme0c0n1.in‚711.0625 disk.dm-0.in‚711.0625$ memory_percent‚97.97203540802002 disk.sda3.in‚0 disk.dm-0.out‚67778.05859375 disk.dm-1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.nvme0n1.in‚711.0625 disk.dm-1.out‚ 236.08984375 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 236.08984375 disk.drbd1777.in‚0 disk.rbd2.out‚136.4970703125* proc.memory.percent‚17.076826095581055 cpu‚25.37084765482566 disk.rbd3.out‚0 disk.rbd1.in‚0" proc.memory.rssMB‚ 2797.8671875õ™†Å‘¥‚(:† óŠäÔøå®™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.557' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1114 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡;½:%¦‚(:š óŠäÔœ disk.sda.out‚ 236.66015625 disk.dm-1.in‚0) disk./.usagePercent‚73.67814682617833 disk.sda.in‚0 disk.rbd1.in‚0$ memory_percent‚97.97148704528809* proc.memory.percent‚17.076826095581055 disk.rbd0.out‚0 disk.rbd1.out‚ 3.12890625 disk.dm-1.out‚ 236.66015625 cpu‚25.36996479251334 disk.sda2.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 332.3515625 proc.cpu.threads‚17 network.sent‚ 1.8717763e+07 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.nvme0c0n1.in‚711.0625( disk.nvme0c0n1.out‚67877.22900390625 disk.drbd1777.out‚0 disk.nvme0n1.in‚711.0625% disk./.usageGB‚1097.8576316833496 disk.sda2.out‚0 disk.sda3.out‚ 236.66015625& disk.nvme0n1.out‚67877.22900390625# disk.dm-0.out‚67877.22900390625 network.recv‚ 2.161966e+06! disk.rbd2.out‚136.58349609375 disk.rbd3.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.rbd3.out‚0 disk.drbd1777.in‚0 disk.dm-0.in‚711.0625" proc.memory.rssMB‚ 2797.8671875 ²›§‚(: ‚‹äÔ휙 gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚92.465' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.493000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.Vw¨‚(:” ‚‹äÔ­àÏœ% disk.nvme0n1.out‚67977.0595703125 disk.dm-0.in‚711.0625 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 237.1015625 network.sent‚ 1.8724001e+07$ disk./.usageGB‚1097.857780456543 disk.dm-1.in‚0 disk.nvme0c0n1.in‚711.0625 disk.sda.in‚0 disk.rbd1.out‚ 3.12890625$ memory_percent‚97.97053337097168" disk.dm-0.out‚67977.0595703125 disk.rbd3.out‚0' disk.nvme0c0n1.out‚67977.0595703125 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.076826095581055 cpu‚25.369196337899854 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚711.0625 disk.rbd0.out‚0! disk.rbd2.out‚136.59521484375' proc.memory.availableMB‚ 332.5078125 network.recv‚ 2.166712e+06 disk.dm-1.out‚ 237.1015625 disk.sda3.in‚0" proc.memory.rssMB‚ 2797.8671875) disk./.usagePercent‚73.67815681047217 disk.sda.out‚ 237.1015625ð!¾Û2©- _wandbruntime‚5926  _runtime‚5926ãß.9ƪÀ & lr-AdamW/pg1‚3.200507792922643e-05 & lr-AdamW/pg2‚3.200507792922643e-05 trainer/global_step‚7949 $ _timestamp‚1.788413315929592e+09 _runtime‚5926.341739185 _step‚650ŒáìË«Å & lr-AdamW/pg1‚3.200507792922643e-05 & lr-AdamW/pg2‚3.200507792922643e-05 trainer/global_step‚7949 $ _timestamp‚1.788413315929592e+09 _runtime‚5926.341739185 _step‚650Šl¸2¬- _wandbruntime‚5926  _runtime‚5926'IŒ ð ­çO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11176947587728 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚235144402921472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚248682314904064 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚5926.342763024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61112356 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45092873 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚237279299205120 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11209249833984 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚248682555720192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11209249833984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106205761 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11176782466324 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚235144402921472 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚237279275265024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.aÝVÉýllocated‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _step‚651 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚223935135084544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160640197 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚224201729247508 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚213024946781184 % _timestamp‚1.7884133159304056e+09 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106205227 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚223935153087488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104696953 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104696953 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11209080834048 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104696951 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚213024963820544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚534 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚224201911408272 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265357188 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160660235 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11403256515072 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235144215918592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61112354 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11403039639040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 265337148 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 trainer/global_step‚7949 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚223935135084544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11209080834048 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235144215918592 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104696951 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160640197 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11209249833984 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚248682555720192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11209249833984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106205761 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11176782466324 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚235144402921472 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚237279275265024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _step‚651 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚223935135084544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160640197 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚224201729247508 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚213024946781184 % _timestamp‚1.7884133159304056e+09 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106205227 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚223935153087488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104696953 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104696953 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11209080834048 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104696951 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚213024963820544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚534 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚224201911408272 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265357188 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160660235 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11403256515072 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235144215918592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61112354 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11403039639040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 265337148 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 trainer/global_step‚7949 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚223935135084544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11209080834048 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235144215918592 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104696951 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160640197 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217‹ÏŒN2¯- _wandbruntime‚5926  _runtime‚5926ƒ½H ÷&°ñM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104709610 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚224229061594388 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265389660 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61119731 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11178171630868 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚652 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160659984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265389660 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106218394 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚224229243755360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚213050889963520 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11210642985472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 trainer/global_step‚7949 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 160680048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11404459607040 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚235173065437696 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11210642985472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61119733 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚235173065437696 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 265369594 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104709612 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106218920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚223962404449280 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45099187 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚248712647820288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45098663 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11404678566912 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104709610 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚235172878421504 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11210473972224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚237308212153344 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11178336752480 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160­Ã™ÀűöM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚235173065437696 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 265369594 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104709612 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106218920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚223962404449280 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45099187 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚248712647820288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45098663 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11404678566912 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104709610 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚235172878421504 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11210473972224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚237308212153344 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11178336752480 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104709610 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚224229061594388 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265389660 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61119731 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11178171630868 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚652 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160659984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265389660 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106218394 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚224229243755360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚213050889963520 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11210642985472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 trainer/global_step‚7949 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 160680048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11404459607040 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚235173065437696 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11210642985472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61119733 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401Œçç¨2²- _wandbruntime‚5926  _runtime‚5926ÔõX³Ò /loss/FRt4/assignment_loss‚2.935120915665346 2loss/SRqqt1/assignment_loss‚2.3213379635260654 2loss/SRqqt4/assignment_loss‚1.3662858191059857 .loss/FRt1/detection_loss‚3.038685598543712 /loss/FRt2/detection_loss‚1.9740246122533625 1loss/SRqqt3/assignment_loss‚1.303183669783175 _runtime‚5926.993446319 /loss/FRt3/detection_loss‚1.9749061541424857 0loss/SRqqt2/detection_loss‚1.323592169640156 /loss/FBt4/detection_loss‚2.4408272832632063 trainer/global_step‚7949 _step‚653 /loss/FRt1/assignment_loss‚4.498566729681833 1loss/FBt2/assignment_loss‚0.46848887909027326 0loss/SRqqt3/detection_loss‚1.722575776837766 epoch‚6 /loss/FRt2/assignment_loss‚2.433102203499187 2loss/SRqqt2/assignment_loss‚1.2243939538796742 1loss/FBt3/assignment_loss‚0.21281278029529135 .loss/FBt1/detection_loss‚3.056970477104187 %loss/total_loss‚46.56040458208167 1loss/FBt4/assignment_loss‚0.49478158036872627 /loss/FBt3/detection_loss‚0.6526671113693211 % _timestamp‚1.7884133165810804e+09 0loss/FRt3/assignment_loss‚2.2889458338419595 0loss/FBt1/assignment_loss‚0.3488308164255264 /loss/FRt4/detection_loss‚1.9775956027648027 1loss/SRqqt1/detection_loss‚2.5822709065217238 0loss/SRqqt4/detection_loss‚2.126472360113772 .loss/FBt2/detection_loss‚3.793965384364128ƒ1 §]´× /loss/FRt4/detection_loss‚1.9775956027648027 1loss/SRqqt1/detection_loss‚2.5822709065217238 0loss/SRqqt4/detection_loss‚2.126472360113772 .loss/FBt2/detection_loss‚3.793965384364128 /loss/FRt4/assignment_loss‚2.935120915665346 2loss/SRqqt1/assignment_loss‚2.3213379635260654 2loss/SRqqt4/assignment_loss‚1.3662858191059857 .loss/FRt1/detection_loss‚3.038685598543712 /loss/FRt2/detection_loss‚1.9740246122533625 1loss/SRqqt3/assignment_loss‚1.303183669783175 _runtime‚5926.993446319 /loss/FRt3/detection_loss‚1.9749061541424857 0loss/SRqqt2/detection_loss‚1.323592169640156 /loss/FBt4/detection_loss‚2.4408272832632063 trainer/global_step‚7949 _step‚653 /loss/FRt1/assignment_loss‚4.498566729681833 1loss/FBt2/assignment_loss‚0.46848887909027326 0loss/SRqqt3/detection_loss‚1.722575776837766 epoch‚6 /loss/FRt2/assignment_loss‚2.433102203499187 2loss/SRqqt2/assignment_loss‚1.2243939538796742 1loss/FBt3/assignment_loss‚0.21281278029529135 .loss/FBt1/detection_loss‚3.056970477104187 %loss/total_loss‚46.56040458208167 1loss/FBt4/assignment_loss‚0.49478158036872627 /loss/FBt3/detection_loss‚0.6526671113693211 % _timestamp‚1.7884133165810804e+09 0loss/FRt3/assignment_loss‚2.2889458338419595 0loss/FBt1/assignment_loss‚0.3488308164255264Ç¿»cœµ‚(:‘ ‘‹äÔþç´™ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.363' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.672600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`òÒ¶‚(:’ ‘‹äÔ솖œ! disk.rbd2.out‚136.65771484375& disk.nvme0c0n1.out‚68143.708984375 disk.sda1.in‚0 disk.dm-1.out‚ 237.8828125$ memory_percent‚97.97396659851074 disk.rbd0.out‚0 network.sent‚ 1.8730113e+07 disk.sda2.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 3.12890625" proc.memory.rssMB‚ 2797.8671875 cpu‚25.368397529493247% disk./.usageGB‚1096.7957763671875$ disk.nvme0n1.out‚68143.708984375 disk.dm-0.in‚711.0625 disk.rbd3.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 237.8828125 disk.drbd1777.in‚0 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 237.8828125 disk.drbd1777.out‚0 disk.nvme0n1.in‚711.0625* proc.memory.percent‚17.076826095581055 network.recv‚ 2.171168e+06) disk./.usagePercent‚73.60688482495475 disk.sda.in‚0 disk.rbd1.in‚0! disk.dm-0.out‚68143.708984375' proc.memory.availableMB‚ 331.9453125 disk.nvme0c0n1.in‚711.0625 disk.rbd0.in‚0k‹&¼œ·‚(:‘  ‹äÔÑæ®™ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.775' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.755000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äÚÚú ¸‚(:•  ‹äÔŠ‹œ% disk./.usageGB‚1096.7957763671875 disk.rbd1.out‚ 3.12890625 disk.dm-0.in‚711.0625! disk.rbd2.out‚138.24755859375 disk.sda2.out‚0 disk.sda3.out‚ 238.234375 network.sent‚ 1.8838032e+07 disk.sda.in‚0 disk.drbd1777.out‚0$ memory_percent‚97.97236919403076" proc.memory.rssMB‚ 2797.8671875* proc.memory.percent‚17.076826095581055& disk.nvme0n1.out‚68144.49658203125 disk.nvme0c0n1.in‚711.0625( disk.nvme0c0n1.out‚68144.49658203125 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 238.234375 cpu‚25.367599249998527) disk./.usagePercent‚73.60688482495475 disk.rbd1.in‚0# disk.dm-0.out‚68144.49658203125 disk.sda2.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 332.20703125 disk.nvme0n1.in‚711.0625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 238.234375 disk.rbd0.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd3.out‚0 network.recv‚ 2.17688e+06´J2¹- _wandbruntime‚5961  _runtime‚5961||ŸEź¿ % lr-AdamW/pg1‚3.13284857150218e-05 % lr-AdamW/pg2‚3.13284857150218e-05 trainer/global_step‚7999 % _timestamp‚1.7884133510072281e+09 _runtime‚5961.419416255 _step‚654÷Ô)ÙÊ»Ä % lr-AdamW/pg1‚3.13284857150218e-05 % lr-AdamW/pg2‚3.13284857150218e-05 trainer/global_step‚7999 % _timestamp‚1.7884133510072281e+09 _runtime‚5961.419416255 _step‚654Žž–•2¼- _wandbruntime‚5961  _runtime‚5961ž/«„í'½çO [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11246254284732 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚250199100816896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 161632899 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11278921156608 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106846882 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 % _timestamp‚1.7884133510079985e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚559 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚238724922676224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11278921156608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _runtime‚5961.420335396 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 161652937 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚236577542480896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚236577542480896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚236577355478016 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚225298603321344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11474154200576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106846323 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61481204 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚225298621324288 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105329903 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 161632899 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105329901 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11278752156672 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11246419406136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚557 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45365119 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚225298621324288 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11278752156672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚238724946616320 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚214322122937344 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217%îLò'¾ìO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106846882 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 % _timestamp‚1.7884133510079985e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚559 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚238724922676224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11278921156608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _runtime‚5961.420335396 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 161652937 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚236577542480896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚236577542480896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚236577355478016 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚225298603321344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11474154200576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106846323 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61481204 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚225298621324288 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105329903 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 161632899 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105329901 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11278752156672 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11246419406136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚557 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45365119 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚225298621324288 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11278752156672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚238724946616320 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚214322122937344 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11246254284732 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚250199100816896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 161632899 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11278921156608 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944šý&é2¿- _wandbruntime‚5962  _runtime‚5962,|ißùÀñM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚236606204997120 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 161652686 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45371152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161672750 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚”Œ±sþ!61488581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 267015312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11280145294848 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106859218 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚250229432687616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚225325872686080 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267015312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45370637 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚225595874690568 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105342562 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11280314308096 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11280145294848 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105342560 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 161652686 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚238753835624448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11475356260352 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚214348049080320 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚515 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 _step‚656 % _timestamp‚1.7884133516592116e+09 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚236606204997120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚517 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11475573123072 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106859735 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _runtime‚5962.071952527 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105342560 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚225325890689024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11247808570888 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105342562 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚250229191884800 trainer/global_step‚7999 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 161672750 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚225325872686080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 266995246 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61488583“˜ÿ‚ü&ÁöM PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105342560 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 161652686 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚238753835624448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11475356260352 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚214348049080320 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚515 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 _step‚656 % _timestamp‚1.7884133516592116e+09 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚236606204997120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚517 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11475573123072 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106859735 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _runtime‚5962.071952527 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105342560 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚225325890689024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11247808570888 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105342562 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚250229191884800 trainer/global_step‚7999 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 161672750 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚225325872686080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 266995246 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61488583 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚236606204997120 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 161652686 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45371152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161672750 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61488581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 267015312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11280145294848 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106859218 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚250229432687616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚225325872686080 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267015312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45370637 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚225595874690568 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105342562 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11280314308096 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11280145294848 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248é·ê2Â- _wandbruntime‚5962  _runtime‚5962ÛŸWÃÑ 0loss/FBt3/assignment_loss‚0.2737627945234899 .loss/FRt2/detection_loss‚2.244214187968861 _runtime‚5962.073386779 0loss/FRt4/assignment_loss‚3.1983002202183592 0loss/FBt2/assignment_loss‚0.4415319909652074 /loss/FRt3/detection_loss‚2.2892393457646274 1loss/SRqqt2/detection_loss‚1.7644486856336394 /loss/FBt3/detection_loss‚0.9316053769208676 /loss/FRt3/assignment_loss‚2.178265087458552 /loss/FRt1/detection_loss‚3.2469057874246077 0loss/SRqqt3/detection_loss‚1.210269518774645 .loss/FBt4/detection_loss‚2.736647571836199 $loss/total_loss‚47.6289524876733 2loss/SRqqt3/assignment_loss‚1.5536487533486094 1loss/FBt4/assignment_loss‚0.33047079119010697 % _timestamp‚1.7884133516610224e+09 .loss/FBt2/detection_loss‚2.787760999467638 trainer/global_step‚7999 _step‚657 /loss/FRt1/assignment_loss‚5.155333085493608 0loss/FRt2/assignment_loss‚2.4538763761520386 .loss/FRt4/detection_loss‚2.091250285798428 0loss/SRqqt4/detection_loss‚1.794704972528944 1loss/SRqqt1/assignment_loss‚2.324077100679278 2loss/SRqqt4/assignment_loss‚1.4550967396182173 1loss/SRqqt1/detection_loss‚2.4466768190264703 /loss/FBt1/detection_loss‚3.0089893905740035 2loss/SRqqt2/assignment_loss‚1.4024066924427947 1loss/FBt1/assignment_loss‚0.30946991386410544 epoch‚6Äâ:×\ÄÖ .loss/FBt2/detection_loss‚2.787760999467638 trainer/global_step‚7999 _step‚657 /loss/FRt1/assignment_loss‚5.155333085493608 0loss/FRt2/assignment_loss‚2.4538763761520386 .loss/FRt4/detection_loss‚2.091250285798428 0loss/SRqqt4/detection_loss‚1.794704972528944 1loss/SRqqt1/assignment_loss‚2.324077100679278 2loss/SRqqt4/assignment_loss‚1.4550967396182173 1loss/SRqqt1/detection_loss‚2.4466768190264703 /loss/FBt1/detection_loss‚3.0089893905740035 2loss/SRqqt2/assignment_loss‚1.4024066924427947 1loss/FBt1/assignment_loss‚0.30946991386410544 epoch‚6 0loss/FBt3/assignment_loss‚0.2737627945234899 .loss/FRt2/detection_loss‚2.244214187968861 _runtime‚5962.073386779 0loss/FRt4/assignment_loss‚3.1983002202183592 0loss/FBt2/assignment_loss‚0.4415319909652074 /loss/FRt3/detection_loss‚2.2892393457646274 1loss/SRqqt2/detection_loss‚1.7644486856336394 /loss/FBt3/detection_loss‚0.9316053769208676 /loss/FRt3/assignment_loss‚2.178265087458552 /loss/FRt1/detection_loss‚3.2469057874246077 0loss/SRqqt3/detection_loss‚1.210269518774645 .loss/FBt4/detection_loss‚2.736647571836199 $loss/total_loss‚47.6289524876733 2loss/SRqqt3/assignment_loss‚1.5536487533486094 1loss/FBt4/assignment_loss‚0.33047079119010697 % _timestamp‚1.7884133516610224e+09‘TB£‘Å‚(:† ¯‹äÔÑ‹“™ gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.339' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÑÛ'Æ‚(:œ ¯‹äÔÜÿ› disk.nvme0n1.in‚711.0625( disk.nvme0c0n1.out‚68144.84033203125# disk.dm-0.out‚68144.84033203125* proc.memory.percent‚17.076826095581055 network.recv‚ 2.180299e+06 disk.nvme0c0n1.in‚711.0625 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.out‚ 238.87890625 disk.sda.out‚ 238.87890625 disk.rbd1.in‚0 disk.rbd0.out‚0$ memory_percent‚97.97251224517822( proc.memory.availableMB‚ 332.18359375% disk./.usageGB‚1096.7957763671875 disk.sda3.in‚0& disk.nvme0n1.out‚68144.84033203125 cpu‚25.366763899397334) disk./.usagePercent‚73.60688482495475 disk.sda2.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.rbd1.out‚ 3.12890625 disk.dm-0.in‚711.0625 network.sent‚ 1.8843452e+07 disk.dm-1.out‚ 238.87890625 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.drbd1777.out‚0 disk.dm-1.in‚0! disk.rbd2.out‚138.37255859375 disk.rbd3.in‚0" proc.memory.rssMB‚ 2797.86718752ÜÈ|Ç‚(:„ ¾‹äÔ龦™ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.64' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô!È‚(:– ¾‹äÔ‹ÖÚœ disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0* proc.memory.percent‚17.076826095581055 proc.cpu.threads‚17 network.sent‚ 1.8952237e+07 disk.rbd3.in‚0" proc.memory.rssMB‚ 2797.8671875" disk./.usageGB‚1096.7958984375 disk.sda1.out‚0 disk.nvme0n1.in‚711.0625 disk.dm-0.in‚711.0625 disk.sda3.out‚ 239.31640625( proc.memory.availableMB‚ 332.17578125 network.recv‚ 2.186904e+06 disk.rbd1.out‚ 3.12890625% disk.nvme0n1.out‚68145.3076171875! disk.rbd2.out‚138.38818359375 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚68145.3076171875) disk./.usagePercent‚73.60689301719584 disk.drbd1777.out‚0 disk.dm-1.out‚ 239.31640625 disk.rbd0.out‚0 disk.drbd1777.in‚0" disk.dm-0.out‚68145.3076171875 disk.rbd3.out‚0 disk.sda.out‚ 239.31640625 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.nvme0c0n1.in‚711.0625$ memory_percent‚97.97255992889404 cpu‚25.365967128228334"¶Äƒ2É- _wandbruntime‚5996  _runtime‚5996¾xÇÊÁ % _timestamp‚1.7884133859154944e+09 _runtime‚5996.327686238 _step‚658 & lr-AdamW/pg1‚3.065589243919995e-05 & lr-AdamW/pg2‚3.065589243919995e-05 trainer/global_step‚8049çÓ 4ÌËÆ & lr-AdamW/pg1‚3.065589243919995e-05 & lr-AdamW/pg2‚3.065589243919995e-05 trainer/global_step‚8049 % _timestamp‚1.7884133859154944e+09 _runtime‚5996.327686238 _step‚658’¾/o\2Ì- _wandbruntime‚5996  _runtime‚5996’­w`ÍæO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45636562 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11544766172672 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 268608492 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚226934991117924 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚215619282054144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238010495037440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 trainer/global_step‚8049 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788413385916288e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 162625601 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107487151 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105962853 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚226662071558144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105962851 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61850056 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11315726103140 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105962853 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚238010682040320 X‘@¶Œ QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11348592479232 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11348423479296 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚238010682040320 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105962851 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107486616 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚240170594027520 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11315891224544 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162645639 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45637095 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚238010495037440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 162625601 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚240170570087424‹=$Xñ'ÎëO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚226662071558144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105962851 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61850056 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11315726103140 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105962853 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚238010682040320 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11348592479232 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11348423479296 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚238010682040320 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105962851 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107486616 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚240170594027520 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11315891224544 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162645639 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45637095 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚238010495037440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 162625601 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚240170570087424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45636562 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11544766172672 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 268608492 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚226934991117924 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚215619282054144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238010495037440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 trainer/global_step‚8049 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788413385916288e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 162625601 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107487151 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105962853 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192“ÊXö2Ï- _wandbruntime‚5996  _runtime‚5996W†«»ö&ÐðM EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚226962505625776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚513 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 268620898 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11349816617472 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11349985630720 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105975510 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61857433 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11349985630720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 268620898 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105975512 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 162665452 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61857431 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 trainer/global_step‚8049 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚226689358925824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚226689358925824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45641910 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 268640964 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚251745900335104 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚215645225236480 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11317115267684 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105975510 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚660 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚238039344556544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚215645208197120 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162645388 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162645388 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 _runtime‚5996.996301475 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 268640964 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107499341 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11546184885248 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11349816617472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚238039344556544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0š5¡’û&ÑõM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 268640964 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚251745900335104 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚215645225236480 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11317115267684 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105975510 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚660 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚238039344556544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚215645208197120 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162645388 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162645388 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 _runtime‚5996.996301475 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 268640964 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107499341 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11546184885248 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11349816617472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚238039344556544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚226962505625776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚513 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 268620898 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11349816617472 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11349985630720 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105975510 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61857433 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11349985630720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 268620898 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105975512 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 162665452 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61857431 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 trainer/global_step‚8049 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚226689358925824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚226689358925824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45641910 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0”5pð2Ò- _wandbruntime‚5996  _runtime‚5996u˃XÓÒ 2loss/SRqqt3/assignment_loss‚1.3490661383435434 _step‚661 2loss/SRqqt2/assignment_loss‚1.1586235215576985 .loss/FRt1/detection_loss‚2.831080024733263 .loss/FBt2/detection_loss‚2.521918870508671 trainer/global_step‚8049 _runtime‚5996.997689319 0loss/FRt2/assignment_loss‚3.0733534379438923 2loss/SRqqt1/assignment_loss‚2.3348651670891307 %loss/total_loss‚46.56185827213679 0loss/FRt4/assignment_loss‚2.8085441794551786 0loss/FBt4/assignment_loss‚0.4497444921871647 -loss/FRt2/detection_loss‚2.14078807099299 1loss/SRqqt1/detection_loss‚2.8288830324359564 1loss/SRqqt2/detection_loss‚1.6541312166800102 .loss/FBt1/detection_loss‚2.813180217376122 /loss/FBt3/detection_loss‚0.7503401552108411 /loss/FRt1/assignment_loss‚5.297751195290509 0loss/FBt3/assignment_loss‚0.2485565829763434 0loss/SRqqt4/detection_loss‚1.772030705852168 epoch‚6 0loss/FRt3/assignment_loss‚2.6110522708994277 2loss/SRqqt4/assignment_loss‚1.6802996276611728 1loss/FBt2/assignment_loss‚0.40368654078338295 /loss/FBt4/detection_loss‚1.9385622441768646 1loss/FBt1/assignment_loss‚0.43420536799777226 .loss/FRt3/detection_loss‚2.069385826587677 0loss/SRqqt3/detection_loss‚1.412091630225887 % _timestamp‚1.7884133865856252e+09 /loss/FRt4/detection_loss‚1.9797177551711191a-å]Ô× 2loss/SRqqt2/assignment_loss‚1.1586235215576985 .loss/FRt1/detection_loss‚2.831080024733263 .loss/FBt2/detection_loss‚2.521918870508671 trainer/global_step‚8049 _runtime‚5996.997689319 0loss/FRt2/assignment_loss‚3.0733534379438923 2loss/SRqqt1/assignment_loss‚2.3348651670891307 %loss/total_loss‚46.56185827213679 0loss/FRt4/assignment_loss‚2.8085441794551786 0loss/FBt4/assignment_loss‚0.4497444921871647 -loss/FRt2/detection_loss‚2.14078807099299 1loss/SRqqt1/detection_loss‚2.8288830324359564 1loss/SRqqt2/detection_loss‚1.6541312166800102 .loss/FBt1/detection_loss‚2.813180217376122 /loss/FBt3/detection_loss‚0.7503401552108411 /loss/FRt1/assignment_loss‚5.297751195290509 0loss/FBt3/assignment_loss‚0.2485565829763434 0loss/SRqqt4/detection_loss‚1.772030705852168 epoch‚6 0loss/FRt3/assignment_loss‚2.6110522708994277 2loss/SRqqt4/assignment_loss‚1.6802996276611728 1loss/FBt2/assignment_loss‚0.40368654078338295 /loss/FBt4/detection_loss‚1.9385622441768646 1loss/FBt1/assignment_loss‚0.43420536799777226 .loss/FRt3/detection_loss‚2.069385826587677 0loss/SRqqt3/detection_loss‚1.412091630225887 % _timestamp‚1.7884133865856252e+09 /loss/FRt4/detection_loss‚1.9797177551711191 2loss/SRqqt3/assignment_loss‚1.3490661383435434 _step‚661•ÀmÕ‚(:… Í‹äÔû¼®™ gpu.0.gpu‚25.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.424' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0848 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ\%ÔÖ‚(:Š Í‹äÔœÙö› network.sent‚ 1.895831e+07 disk.sda2.out‚0) disk./.usagePercent‚73.60689634529378 disk.rbd3.in‚0 disk.sda3.out‚ 239.859375 disk.sda.in‚0 disk.nvme0n1.in‚711.0625% disk.nvme0n1.out‚68145.6396484375 disk.rbd2.out‚ 138.474609375' disk.nvme0c0n1.out‚68145.6396484375 disk.sda3.in‚0$ memory_percent‚97.97592163085938 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚711.0625" disk.dm-0.out‚68145.6396484375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚711.0625 disk.sda.out‚ 239.859375 disk.drbd1777.in‚0 proc.cpu.threads‚17 network.recv‚ 2.19161e+06 disk.rbd1.out‚ 3.12890625 disk.dm-1.out‚ 239.859375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.out‚0# proc.memory.availableMB‚331.625 cpu‚25.365187180100104 disk.rbd2.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.076826095581055% disk./.usageGB‚1096.7959480285645 disk.rbd0.in‚0 disk.rbd3.out‚0" proc.memory.rssMB‚ 2797.8671875Œ…ÇMœ×‚(:‘ Ü‹äÔʹ™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚112.533' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.506600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ[1. Ø‚(:• Ü‹äÔž‡ë› network.recv‚ 2.197218e+06" disk.dm-0.out‚68146.3193359375 proc.cpu.threads‚17 disk.nvme0n1.in‚711.0625 disk.dm-0.in‚711.0625 disk.sda.in‚0 network.sent‚ 1.9066224e+07 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 138.490234375 disk.nvme0c0n1.in‚711.0625 disk.sda.out‚ 240.42578125 disk.dm-1.in‚0 disk.sda3.out‚ 240.42578125# memory_percent‚97.9729413986206 disk.rbd1.out‚ 3.12890625 disk.rbd2.in‚0 disk.rbd3.in‚0' disk.nvme0c0n1.out‚68146.3193359375 disk.sda2.in‚0 disk.sda3.in‚0* proc.memory.percent‚17.076826095581055% disk./.usageGB‚1096.7959327697754 disk.drbd1777.out‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 332.11328125" proc.memory.rssMB‚ 2797.8671875) disk./.usagePercent‚73.60689532126365 disk.sda1.out‚0 disk.drbd1777.in‚0% disk.nvme0n1.out‚68146.3193359375 disk.dm-1.out‚ 240.42578125 disk.rbd0.in‚0 disk.sda2.out‚0 cpu‚25.36440517277828 disk.rbd3.out‚0ð=Ï¢‘Ù‚(:† ë‹äÔÄØÅ™ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.602' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3204 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VpóÚ‚(: ë‹äÔîéÙ› disk.sda1.in‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚711.0625 cpu‚25.36358612341164 network.recv‚ 2.20069e+06 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0$ memory_percent‚97.97301292419434 network.sent‚ 1.907184e+07$ disk./.usageGB‚1096.796012878418 disk.sda3.out‚ 240.890625 disk.rbd1.in‚0& disk.nvme0n1.out‚68146.98486328125# disk.dm-0.out‚68146.98486328125 disk.rbd3.out‚0 disk.drbd1777.out‚0 disk.rbd2.in‚0( disk.nvme0c0n1.out‚68146.98486328125 disk.sda1.out‚0) disk./.usagePercent‚73.60690069742188 disk.rbd0.in‚0 disk.rbd2.out‚ 138.615234375 disk.sda.in‚0 disk.dm-0.in‚711.0625 disk.nvme0c0n1.in‚711.0625 disk.sda.out‚ 240.890625 disk.rbd1.out‚ 3.12890625' proc.memory.availableMB‚ 332.1015625" proc.memory.rssMB‚ 2797.8671875 proc.cpu.threads‚17* proc.memory.percent‚17.076826095581055 disk.dm-1.in‚0 disk.dm-1.out‚ 240.890625 disk.sda2.out‚0 disk.sda3.in‚0©_2Û- _wandbruntime‚6031  _runtime‚6031®¤ÈÜ ' lr-AdamW/pg1‚2.9987438269191463e-05 ' lr-AdamW/pg2‚2.9987438269191463e-05 trainer/global_step‚8099 % _timestamp‚1.7884134206404877e+09 _runtime‚ 6031.05263604 _step‚662žÂPôÍÝÇ ' lr-AdamW/pg1‚2.9987438269191463e-05 ' lr-AdamW/pg2‚2.9987438269191463e-05 trainer/global_step‚8099 % _timestamp‚1.7884134206404877e+09 _runtime‚ 6031.05263604 _step‚662–‰—€2Þ- _wandbruntime‚6031  _runtime‚6031’(Õì'ßæO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 270214104 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106595803 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 163618303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163638341 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11615839387648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 270234144 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106595803 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚239443634596864 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108126158 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45907798 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163638341 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚228025557797888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚239443821599744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚228301804213896 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11418094801920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11418263801856 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚253231850496000 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108126704 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 trainer/global_step‚8099 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚239443634596864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11418094801920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45907254 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11385197921548 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚228025557797888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 % _timestamp‚1.7884134206412332e+09 _runtime‚ 6031.05352773 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚241616241438720 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106595803 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚239443634596864 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108126158 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45907798 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163638341 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚228025557797888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚239443821599744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚228301804213896 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11418094801920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11418263801856 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚253231850496000 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108126704 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 trainer/global_step‚8099 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚239443634596864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11418094801920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45907254 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11385197921548 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚228025557797888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 % _timestamp‚1.7884134206412332e+09 _runtime‚ 6031.05352773 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚241616241438720 BI2á- _wandbruntime‚6031  _runtime‚6031Kð¬5ÇâðM SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106608462 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11419656953344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163658154 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108139442 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚239472484115968 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11419487940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚228329136560984 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚216942367313920 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚216942384353280 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 163638090 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 106608462 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 $ _timestamp‚1.788413421283311e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 _runtime‚6031.695904721 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 270246550 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚239472297099776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚228052827162624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11419656953344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚253262180579840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108138905 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62226281 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚228052809159680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11617050132992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11386587086092 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163638090 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241645154386944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11617260704256 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 270266616~ïxû&ãõM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62226283 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 270266616 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106608462 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11419656953344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163658154 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108139442 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚239472484115968 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11419487940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚228329136560984 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚216942367313920 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚216942384353280 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 163638090 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 106608462 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 $ _timestamp‚1.788413421283311e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 _runtime‚6031.695904721 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 270246550 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚239472297099776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚228052827162624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11419656953344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚253262180579840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108138905 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62226281 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚228052809159680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11617050132992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11386587086092 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163638090 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241645154386944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11617260704256 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108767375 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚229389008031744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11686497684992 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218213600287744 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 271839756 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 trainer/global_step‚8149 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164611005 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚501 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11686708269568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚240876774156288 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108766874 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46179120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107228753 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62587754 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164631043 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229389008031744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271859796 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164611005 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11454834861360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 $ _timestamp‚1.788413455163876e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11487766124544 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107228751 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚243061864909824 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11454669739956 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚229389026034688 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚240876774156288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62587756 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11487935124480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11487766124544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚229389026034688 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚254748597119488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JDeviceStatsMonitor.on_t«Bqv¤$rain_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108767375 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚229389008031744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11686497684992 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218213600287744 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 271839756 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 trainer/global_step‚8149 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164611005 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚501 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11686708269568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚240876774156288 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108766874 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46179120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107228753 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62587754 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164631043 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229389008031744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271859796 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164611005 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11454834861360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 $ _timestamp‚1.788413455163876e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11487766124544 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107228751 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚243061864909824 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11454669739956 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚229389026034688 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚240876774156288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62587756 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11487935124480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11487766124544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚229389026034688 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚254748597119488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 271892268 trainer/global_step‚8149 _step‚668 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚240905623675392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚218239526430720 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11456224026112 _runtime‚6066.231581881 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164650856 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚476 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚243090801798144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 271892268 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚229416277396480 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚229416277396480 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚229416295399424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚229695767496192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 271872202 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚254778925741056 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11688123942912 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11687915468800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107241412 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚478 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107241410 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11489159262720 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚218239543470080 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108780280 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11489328275968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46184671 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107241410 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164630792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚229695585335220 % _timestamp‚1.7884134558190632e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62595131 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108779802 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160ó3Ûôü&óöM H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107241410 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164630792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚229695585335220 % _timestamp‚1.7884134558190632e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62595131 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108779802 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚240905623675392 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 271892268 trainer/global_step‚8149 _step‚668 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚240905623675392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚218239526430720 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11456224026112 _runtime‚6066.231581881 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164650856 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚476 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚243090801798144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 271892268 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚229416277396480 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚229416277396480 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚229416295399424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚229695767496192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 271872202 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚254778925741056 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11688123942912 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11687915468800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107241412 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚478 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107241410 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11489159262720 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚218239543470080 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108780280 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11489328275968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46184671 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128œ\^2ô- _wandbruntime‚6066  _runtime‚6066õIK›UõÏ 1loss/SRqqt4/assignment_loss‚1.568615114417943 /loss/FRt1/detection_loss‚2.9548695729329038 0loss/FRt4/assignment_loss‚3.1123829416508944 1loss/SRqqt3/assignment_loss‚1.524811907105946 0loss/SRqqt3/detection_loss‚1.447103947667139 -loss/FBt2/detection_loss‚2.07886666059494 _step‚669 /loss/FRt1/assignment_loss‚4.451180338859558 0loss/FBt1/assignment_loss‚0.3140944473730501 1loss/SRqqt4/detection_loss‚1.6328608412634242 /loss/FBt3/detection_loss‚0.7584490965552894 % _timestamp‚1.7884134558207035e+09 2loss/SRqqt1/assignment_loss‚1.9445845817818361 1loss/SRqqt2/assignment_loss‚1.124596572809798 0loss/FBt2/assignment_loss‚0.3737803911149967 0loss/FBt4/assignment_loss‚0.6494504309298463 .loss/FBt1/detection_loss‚2.598248638890006 0loss/FRt3/assignment_loss‚1.8027424866502935 .loss/FRt4/detection_loss‚2.116620808839798 0loss/SRqqt1/detection_loss‚2.647923280211056 1loss/SRqqt2/detection_loss‚1.5656590569106972 .loss/FBt4/detection_loss‚3.408536081512769 trainer/global_step‚8149 _runtime‚6066.232732659 /loss/FRt2/detection_loss‚2.1061090171337127 /loss/FRt3/detection_loss‚1.6813485377214172 epoch‚6 0loss/FRt2/assignment_loss‚3.1977984523773193 1loss/FBt3/assignment_loss‚0.16181855886129598 %loss/total_loss‚45.222451764165947EáZöÔ /loss/FRt1/assignment_loss‚4.451180338859558 0loss/FBt1/assignment_loss‚0.3140944473730501 1loss/SRqqt4/detection_loss‚1.6328608412634242 /loss/FBt3/detection_loss‚0.7584490965552894 % _timestamp‚1.7884134558207035e+09 2loss/SRqqt1/assignment_loss‚1.9445845817818361 1loss/SRqqt2/assignment_loss‚1.124596572809798 0loss/FBt2/assignment_loss‚0.3737803911149967 0loss/FBt4/assignment_loss‚0.6494504309298463 .loss/FBt1/detection_loss‚2.598248638890006 0loss/FRt3/assignment_loss‚1.8027424866502935 .loss/FRt4/detection_loss‚2.116620808839798 0loss/SRqqt1/detection_loss‚2.647923280211056 1loss/SRqqt2/detection_loss‚1.5656590569106972 .loss/FBt4/detection_loss‚3.408536081512769 trainer/global_step‚8149 _runtime‚6066.232732659 /loss/FRt2/detection_loss‚2.1061090171337127 /loss/FRt3/detection_loss‚1.6813485377214172 epoch‚6 0loss/FRt2/assignment_loss‚3.1977984523773193 1loss/FBt3/assignment_loss‚0.16181855886129598 %loss/total_loss‚45.22245176416594 1loss/SRqqt4/assignment_loss‚1.568615114417943 /loss/FRt1/detection_loss‚2.9548695729329038 0loss/FRt4/assignment_loss‚3.1123829416508944 1loss/SRqqt3/assignment_loss‚1.524811907105946 0loss/SRqqt3/detection_loss‚1.447103947667139 -loss/FBt2/detection_loss‚2.07886666059494 _step‚669o¤Ð½›÷‚(: ˜ŒäÔ–äÕ› gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚81.797' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚16.359399999999997 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Oc<ý|ø‚(:˜ ˜ŒäÔ•Þ$ disk./.usageGB‚1096.625099182129 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd2.out‚ ÓÁÀg§140.345703125 network.recv‚ 2.215789e+06 disk.sda3.out‚ 242.30859375 disk.dm-1.out‚ 242.30859375 disk.nvme0c0n1.in‚711.0625 disk.sda3.in‚0 disk.sda.in‚0 disk.rbd1.out‚ 3.21484375# disk.dm-0.out‚68150.48486328125 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚711.0625 network.sent‚ 1.9191979e+07 disk.sda2.out‚0 disk.sda.out‚ 242.30859375 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.076826095581055 disk.rbd2.in‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚68150.48486328125" proc.memory.rssMB‚ 2797.8671875 cpu‚25.36125507540401) disk./.usagePercent‚73.59543053585776& disk.nvme0n1.out‚68150.48486328125 disk.rbd3.out‚0$ memory_percent‚97.97341823577881 disk.nvme0n1.in‚711.0625( proc.memory.availableMB‚ 332.03515625UUÂÇ‘ù‚(:† §ŒäÔ“äž™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.158' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4316 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž¶$‹ú‚(: §ŒäÔö‹Ô› network.recv‚ 2.221257e+06& disk.nvme0n1.out‚68283.96142578125 disk.dm-1.in‚0 disk.nvme0c0n1.in‚711.0625 disk.sda1.out‚0* proc.memory.percent‚17.076826095581055 disk.sda.in‚0) disk./.usagePercent‚73.61088033454163 disk.rbd3.out‚0( disk.nvme0c0n1.out‚68283.96142578125$ memory_percent‚97.97515869140625" proc.memory.rssMB‚ 2797.8671875 proc.cpu.threads‚17 network.sent‚ 1.9300023e+07# disk.dm-0.out‚68283.96142578125 disk.dm-1.out‚ 242.890625 disk.rbd2.in‚0 disk.sda.out‚ 242.890625 disk.dm-0.in‚711.0625$ disk./.usageGB‚1096.855312347412 disk.rbd1.out‚ 3.27734375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1777.in‚0" proc.memory.availableMB‚331.75 disk.nvme0n1.in‚711.0625! disk.rbd2.out‚140.43212890625 disk.sda2.in‚0 disk.sda3.out‚ 242.890625 disk.sda1.in‚0 disk.drbd1777.out‚0 cpu‚25.360509449163228 disk.rbd1.in‚0 disk.rbd3.in‚0©dá2û- _wandbruntime‚6100  _runtime‚6100ÍcìÉüà ' lr-AdamW/pg1‚2.8663503574388707e-05 ' lr-AdamW/pg2‚2.8663503574388707e-05 trainer/global_step‚8199 % _timestamp‚1.7884134902717698e+09 _runtime‚6100.683884324 _step‚670¡ºÑ‚ÎýÈ ' lr-AdamW/pg1‚2.8663503574388707e-05 ' lr-AdamW/pg2‚2.8663503574388707e-05 trainer/global_step‚8199 % _timestamp‚1.7884134902717698e+09 _runtime‚6100.683884324 _step‚670ž5ýµ>2þ- _wandbruntime‚6100  _runtime‚6100ĶGí'ÿçO UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 165623745 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚231035066084312 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11557437447168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚242310100718592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚6100.684775422 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242309913715712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚244507512321024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11557437447168 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 165623745 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109408796 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109408292 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46452190 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚230752494271488 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107861703 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 165603707 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62956604 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11557606447104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11524306679768 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚242309913715712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚230752476268544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206390272 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11757553625088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11557606447104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230330368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 trainer/global_step‚8199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚219510759404544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚231034883923548 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46451688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884134902724788e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚256265089886208 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11524141558364 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚502 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 165603707 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚244507536261120 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11757347234816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚671 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107861701 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚230752476268544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107861703 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4107120642Ç ôò'€ ìO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚231034883923548 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46451688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884134902724788e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚256265089886208 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11524141558364 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚502 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 165603707 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚244507536261120 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11757347234816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚671 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107861701 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚230752476268544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107861703 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 165623745 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚231035066084312 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11557437447168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚242310100718592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚6100.684775422 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242309913715712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚244507512321024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11557437447168 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 165623745 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109408796 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109408292 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46452190 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚230752494271488 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107861703 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 165603707 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62956604 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11557606447104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11524306679768 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚242309913715712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚230752476268544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206390272 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11757553625088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11557606447104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230330368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 trainer/global_step‚8199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚219510759404544Ÿ‚ækÚ2 - _wandbruntime‚6101  _runtime‚61014ÒU$‚ ñM bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11758977919488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107874362 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚256295427128832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚219536702586880 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62963983 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165643558 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165623494 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11758771542528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚219536685547520 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _step‚672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11558830585344 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 165643558 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109421522 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚231062216270428 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884134909211938e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46457539 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 165623494 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚230779763636224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46457038 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107874360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚242338576218624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚230779745633280 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109421019 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚503 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚242338763234816 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 273517920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11558999598592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62963981 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚244536449209344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11558999598592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11525695844520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚230779763636224 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚230779745633280 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273517920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 273497854 trainer/global_step‚8199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244536425269248 XJDeviceStatsMonitor.on_train_batch_end/inactive_sÞ܃á¢plit_bytes.large_pool.peak‚ 104436224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107874360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11558830585344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚242338763234816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚501>ª'ü&ƒ öM FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚503 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚242338763234816 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 273517920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11558999598592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62963981 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚244536449209344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11558999598592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11525695844520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚230779763636224 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚230779745633280 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273517920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 273497854 trainer/global_step‚8199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244536425269248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107874360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11558830585344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚242338763234816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚501 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11758977919488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107874362 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚256295427128832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚219536702586880 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62963983 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165643558 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165623494 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11758771542528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚219536685547520 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _step‚672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11558830585344 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 165643558 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109421522 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚231062216270428 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884134909211938e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46457539 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 165623494 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚230779763636224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46457038 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107874360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚242338576218624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚230779745633280 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109421019 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 % _timestamp‚1.7884135255778153e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108494653 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚257781609847296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚232115962508288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46722520 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11828213493248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11627108769792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚232401514858756 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166616447 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚537 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108494651 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108494653 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 166616447 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚232401697019520 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11627277769728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11593613376772 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚243743053275136 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110048513 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110047974 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚245953159732224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚675 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚243743053275136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11593778498176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 trainer/global_step‚8249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 166596409 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚232115944505344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚243743240278016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63325456 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11828426174976 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108494651 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166596409 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚6135.990228284 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚243743240278016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚243743053275136 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110048513 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110047974 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚245953159732224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚675 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚243743053275136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11593778498176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 trainer/global_step‚8249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 166596409 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚232115944505344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚243743240278016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63325456 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11828426174976 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108494651 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166596409 GEM I?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚6135.990228284 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚243743240278016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 % _timestamp‚1.7884135255778153e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108494653 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚257781609847296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚232115962508288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46722520 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11828213493248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11627108769792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚232401514858756 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166616447 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚537 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0£ˆ€¨2“ - _wandbruntime‚6136  _runtime‚6136\#}÷&” ñM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 275123506 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚257811937584128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚232428847205636 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108507310 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108507310 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 166636260 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166616196 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11829626198016 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884135262316062e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108507312 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚676 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚243771902794240 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110060720 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108507312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46728409 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚220833861703680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11595002541316 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚232143231873024 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275143572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63332833 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 275143572 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚220833844664320 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11628670921216 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 275123506 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63332831 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚257811698878464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 166636260 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110061242 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚6136.644362373 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11628501907968 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚232429029366608 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0íþ0”ü&• öM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚232143231873024 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275143572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63332833 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 275143572 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚220833844664320 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11628670921216 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 275123506 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63332831 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚257811698878464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 166636260 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110061242 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚6136.644362373 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11628501907968 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚232429029366608 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 275123506 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚257811937584128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚232428847205636 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108507310 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108507310 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 166636260 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166616196 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11829626198016 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884135262316062e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108507312 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚676 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚243771902794240 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110060720 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108507312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46728409 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚220833861703680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11595002541316 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408¤­w2– - _wandbruntime‚6136  _runtime‚6136ëíZ— Ô /loss/FBt1/detection_loss‚2.7099979274413166 1loss/SRqqt4/assignment_loss‚1.582660443403504 .loss/FRt1/detection_loss‚3.371674358844757 /loss/FRt4/detection_loss‚2.1781935612005845 1loss/SRqqt4/detection_loss‚1.7629922906106168 /loss/FBt3/detection_loss‚0.7081305596358632 % _timestamp‚1.7884135262336924e+09 0loss/FRt2/assignment_loss‚3.2818911174933114 /loss/FBt2/detection_loss‚2.8754206895828247 /loss/FRt1/assignment_loss‚4.368312705646861 /loss/FRt3/assignment_loss‚2.125726780375919 0loss/FBt4/assignment_loss‚0.6235755745914172 2loss/SRqqt3/assignment_loss‚1.4765236882840174 1loss/FBt3/assignment_loss‚0.21329946738121394 /loss/FRt2/detection_loss‚2.1257996045880847 /loss/SRqqt1/detection_loss‚2.65948657478605 1loss/SRqqt3/detection_loss‚1.3551693562481364 _runtime‚6136.645727574 /loss/FRt4/assignment_loss‚3.427335295294012 2loss/SRqqt1/assignment_loss‚2.6548466150249754 1loss/FBt1/assignment_loss‚0.45572200323070267 1loss/FBt2/assignment_loss‚0.22220310707659033 0loss/SRqqt2/detection_loss‚1.421339192625248 _step‚677 epoch‚6 trainer/global_step‚8249 2loss/SRqqt2/assignment_loss‚1.5237896943634206 /loss/FRt3/detection_loss‚2.6921029678873114 /loss/FBt4/detection_loss‚2.4983167431571265 %loss/total_loss‚48.31451031877387–_¬á_˜ Ù 1loss/FBt2/assignment_loss‚0.22220310707659033 0loss/SRqqt2/detection_loss‚1.421339192625248 _step‚677 epoch‚6 trainer/global_step‚8249 2loss/SRqqt2/assignment_loss‚1.5237896943634206 /loss/FRt3/detection_loss‚2.6921029678873114 /loss/FBt4/detection_loss‚2.4983167431571265 %loss/total_loss‚48.31451031877387 /loss/FBt1/detection_loss‚2.7099979274413166 1loss/SRqqt4/assignment_loss‚1.582660443403504 .loss/FRt1/detection_loss‚3.371674358844757 /loss/FRt4/detection_loss‚2.1781935612005845 1loss/SRqqt4/detection_loss‚1.7629922906106168 /loss/FBt3/detection_loss‚0.7081305596358632 % _timestamp‚1.7884135262336924e+09 0loss/FRt2/assignment_loss‚3.2818911174933114 /loss/FBt2/detection_loss‚2.8754206895828247 /loss/FRt1/assignment_loss‚4.368312705646861 /loss/FRt3/assignment_loss‚2.125726780375919 0loss/FBt4/assignment_loss‚0.6235755745914172 2loss/SRqqt3/assignment_loss‚1.4765236882840174 1loss/FBt3/assignment_loss‚0.21329946738121394 /loss/FRt2/detection_loss‚2.1257996045880847 /loss/SRqqt1/detection_loss‚2.65948657478605 1loss/SRqqt3/detection_loss‚1.3551693562481364 _runtime‚6136.645727574 /loss/FRt4/assignment_loss‚3.427335295294012 2loss/SRqqt1/assignment_loss‚2.6548466150249754 1loss/FBt1/assignment_loss‚0.45572200323070267¥MÒ ›™ ‚(: ãŒäÔâÅÇ™ gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.993' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.798599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆè<ª%š ‚(:š ãŒäÔ¢¸„œ$ disk./.usageGB‚1096.885326385498 disk.rbd3.in‚0 disk.sda1.out‚0 disk.drbd1777.out‚0* proc.memory.percent‚17.076826095581055( disk.nvme0c0n1.out‚68419.20068359375) disk./.usagePercent‚73.61289460182104 disk.nvme0c0n1.in‚711.0625 proc.cpu.threads‚17! disk.rbd2.out‚142.16259765625 disk.rbd3.out‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚711.0625$ memory_percent‚97.97403812408447 cpu‚25.35744851116286 disk.dm-1.out‚ 245.1015625 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda1.in‚0& disk.nvme0n1.out‚68419.20068359375 disk.rbd2.in‚0 disk.sda.in‚0 disk.sda.out‚ 245.1015625 disk.rbd1.out‚ 3.37255859375( proc.memory.availableMB‚ 331.93359375 network.sent‚ 1.9425602e+07 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 245.1015625 disk.dm-0.in‚711.0625# disk.dm-0.out‚68419.20068359375" proc.memory.rssMB‚ 2797.8671875 network.recv‚ 2.239134e+06 disk.sda2.out‚0‚~j‘› ‚(:† òŒäԠت™ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.291' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4582 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦OÐ_'œ ‚(:œ òŒäÔƒ·Ýœ disk.rbd1.in‚0 disk.rbd1.out‚ 3.37255859375 disk.dm-0.in‚711.0625 disk.rbd0.in‚0) disk./.usagePercent‚73.61343477771828! disk.rbd2.out‚142.16259765625 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda.out‚ 245.4453125 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚711.0625 disk.dm-1.in‚0$ memory_percent‚97.97403812408447 proc.cpu.threads‚17 disk.sda.in‚0* proc.memory.percent‚17.076826095581055 network.recv‚ 2.246066e+06% disk./.usageGB‚1096.8933753967285 disk.sda3.out‚ 245.4453125 disk.drbd1777.out‚0 disk.nvme0n1.in‚711.0625 cpu‚25.356737941008877 network.sent‚ 1.9534726e+07 disk.rbd3.out‚0& disk.nvme0n1.out‚68427.85107421875 disk.rbd2.in‚0( disk.nvme0c0n1.out‚68427.85107421875 disk.sda1.out‚0# disk.dm-0.out‚68427.85107421875 disk.dm-1.out‚ 245.4453125( proc.memory.availableMB‚ 331.93359375" proc.memory.rssMB‚ 2797.8671875 disk.sda2.in‚0!ú‡2 - _wandbruntime‚6170  _runtime‚6170¸“ˆÈž  $ _timestamp‚1.788413560584866e+09 _runtime‚6170.997136088 _step‚678 ' lr-AdamW/pg1‚2.7357785197139288e-05 ' lr-AdamW/pg2‚2.7357785197139288e-05 trainer/global_step‚8299Õ3ؽÍŸ Ç _runtime‚6170.997136088 _step‚678 ' lr-AdamW/pg1‚2.7357785197139288e-05 ' lr-AdamW/pg2‚2.7357785197139288e-05 trainer/global_step‚8299 $ _timestamp‚1.788413560584866e+09¦elû‹2  - _wandbruntime‚6170  _runtime‚6170Ë×6 ¡ æO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11663085195180 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚29ºeã&45176379837440 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚233479412742144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63694304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 167589111 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚259298146398208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _step‚679 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11696780092416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11696780092416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46994524 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚233479430745088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚247398807143424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚233479412742144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167609149 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110689379 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110688828 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚222105060598784 $ _timestamp‚1.788413560585666e+09 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109127603 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 167609149 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167589111 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276736752 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚245176379837440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚259297903484928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11899315314688 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚245176192834560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11899096341504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚259297903484928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11899315314688 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚245176192834560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11899096341504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚233479430745088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚247398807143424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚233479412742144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167609149 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110689379 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110688828 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚222105060598784 $ _timestamp‚1.788413560585666e+09 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109127603 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 167609149 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167589111 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276736752 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0§ÁÓ2£ - _wandbruntime‚6171  _runtime‚6171'Gö÷&¤ ñM VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47000709 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚680 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109140262 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167628962 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚233506700109824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63701683 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚222131020820480 % _timestamp‚1.7884135612462587e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109140260 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167608898 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11698173230592 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 167608898 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276769224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11900725715456 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11664474359724 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚233506682106880 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11698173230592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚259328469747200 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109140262 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11900515144192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚233795478140844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 276749158 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47000175 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109140260 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110701856 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11664639481336 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 _runtime‚6171.658851618 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 167628962 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 276769224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110702392 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚8299 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11698342243840 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚233795660301816 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚222131003781120 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚233506700109824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63701683 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚222131020820480 % _timestamp‚1.7884135612462587e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109140260 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167608898 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11698173230592 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 167608898 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276769224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11900725715456 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11664474359724 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚233506682106880 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11698173230592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚259328469747200 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109140262 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11900515144192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚233795478140844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 276749158 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47000175 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109140260 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110701856 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11664639481336 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 _runtime‚6171.658851618¨]‰© 2¦ - _wandbruntime‚6171  _runtime‚6171¾z•'U§ Ï /loss/FRt2/detection_loss‚2.2253136688724475 /loss/FRt4/detection_loss‚2.3409278302089027 1loss/SRqqt1/assignment_loss‚2.156972510474069 0loss/FBt3/assignment_loss‚0.2893818535039857 0loss/FBt4/assignment_loss‚0.6406881295144558 /loss/FRt3/detection_loss‚1.8595570427676043 .loss/FBt1/detection_loss‚2.882368015391486 1loss/SRqqt3/detection_loss‚1.3889559734808772 &loss/total_loss‚47.108731391291194 epoch‚6 _step‚681 /loss/FRt2/assignment_loss‚3.269923250725929 .loss/FRt1/detection_loss‚3.069311818251243 /loss/FBt3/detection_loss‚0.6812398379883304 .loss/FBt4/detection_loss‚2.640761026740074 /loss/FRt4/assignment_loss‚3.363178382749143 2loss/SRqqt4/assignment_loss‚1.7422382615084917 0loss/FBt1/assignment_loss‚0.7062142532438104 % _timestamp‚1.7884135612483048e+09 0loss/FBt2/assignment_loss‚0.5943372217672211 0loss/SRqqt1/detection_loss‚2.700970321893692 0loss/SRqqt2/detection_loss‚1.396416860918204 1loss/SRqqt4/detection_loss‚1.7510321348343256 .loss/FBt2/detection_loss‚2.764567571026938 0loss/FRt3/assignment_loss‚1.7294883797566096 2loss/SRqqt3/assignment_loss‚1.3080163632045712 trainer/global_step‚8299 _runtime‚6171.660380949 1loss/SRqqt2/assignment_loss‚1.402166388630867 /loss/FRt1/assignment_loss‚4.204704293837914’¢ÍZ¨ Ô 0loss/FBt2/assignment_loss‚0.5943372217672211 0loss/SRqqt1/detection_loss‚2.700970321893692 0loss/SRqqt2/detection_loss‚1.396416860918204 1loss/SRqqt4/detection_loss‚1.7510321348343256 .loss/FBt2/detection_loss‚2.764567571026938 0loss/FRt3/assignment_loss‚1.7294883797566096 2loss/SRqqt3/assignment_loss‚1.3080163632045712 trainer/global_step‚8299 _runtime‚6171.660380949 1loss/SRqqt2/assignment_loss‚1.402166388630867 /loss/FRt1/assignment_loss‚4.204704293837914 /loss/FRt2/detection_loss‚2.2253136688724475 /loss/FRt4/detection_loss‚2.3409278302089027 1loss/SRqqt1/assignment_loss‚2.156972510474069 0loss/FBt3/assignment_loss‚0.2893818535039857 0loss/FBt4/assignment_loss‚0.6406881295144558 /loss/FRt3/detection_loss‚1.8595570427676043 .loss/FBt1/detection_loss‚2.882368015391486 1loss/SRqqt3/detection_loss‚1.3889559734808772 &loss/total_loss‚47.108731391291194 epoch‚6 _step‚681 /loss/FRt2/assignment_loss‚3.269923250725929 .loss/FRt1/detection_loss‚3.069311818251243 /loss/FBt3/detection_loss‚0.6812398379883304 .loss/FBt4/detection_loss‚2.640761026740074 /loss/FRt4/assignment_loss‚3.363178382749143 2loss/SRqqt4/assignment_loss‚1.7422382615084917 0loss/FBt1/assignment_loss‚0.7062142532438104 % _timestamp‚1.7884135612483048e+09©(«ª© ‚(:… äÔ½©¾™ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚94.901' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.9802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[GD#&ª ‚(:› äԛ disk.nvme0c0n1.in‚711.0625 disk.sda2.in‚0 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0$ memory_percent‚97.97742366790771* proc.memory.percent‚17.076826095581055 disk.nvme0n1.in‚711.0625 disk.dm-0.in‚711.0625# disk.dm-0.out‚68428.24169921875 disk.sda.out‚ 246.04296875 disk.sda1.in‚0 cpu‚25.35599841624092 disk.dm-1.out‚ 246.04296875 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 3.435546875 network.recv‚ 2.250563e+06& disk.nvme0n1.out‚68428.24169921875( disk.nvme0c0n1.out‚68428.24169921875 proc.cpu.threads‚17 disk.sda1.out‚0 network.sent‚ 1.9540847e+07% disk./.usageGB‚1096.8933792114258 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 246.04296875 disk.sda.in‚0" proc.memory.rssMB‚ 2797.8671875 disk.rbd2.out‚142.2607421875 disk.sda2.out‚0( proc.memory.availableMB‚ 331.37890625) disk./.usagePercent‚73.61343503372582 disk.rbd0.in‚0 disk.drbd1777.in‚0±}ýQ« ‚(:„ äÔŠåÄ™ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.028 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P(½¬ ‚(:’ äÔ¸·“œ network.sent‚ 1.9648665e+07% disk./.usageGB‚1096.8919219970703 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 246.3984375 disk.rbd2.out‚142.2607421875 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚711.0625 disk.dm-1.out‚ 246.3984375 disk.rbd0.in‚0 disk.rbd3.in‚0" proc.memory.rssMB‚ 2797.8671875 network.recv‚ 2.256261e+06 disk.sda1.out‚0 disk.rbd1.out‚ 3.435546875 disk.nvme0n1.in‚711.0625$ disk.nvme0n1.out‚68428.755859375 disk.rbd0.out‚0 disk.rbd2.in‚0& disk.nvme0c0n1.out‚68428.755859375) disk./.usagePercent‚73.61333723884775 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.sda3.in‚0' proc.memory.availableMB‚ 331.6015625* proc.memory.percent‚17.076826095581055 cpu‚25.355263184094206 disk.drbd1777.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 246.3984375 proc.cpu.threads‚17! disk.dm-0.out‚68428.755859375 disk.nvme0c0n1.in‚711.0625$ memory_percent‚97.97606468200684ñ¿Ï³2­ - _wandbruntime‚6205  _runtime‚6205ëf;ÕÇ® Á & lr-AdamW/pg1‚2.671209786509736e-05 & lr-AdamW/pg2‚2.671209786509736e-05 trainer/global_step‚8349 % _timestamp‚1.7884135953766208e+09 _runtime‚6205.788764006 _step‚682B>Ó̯ Æ & lr-AdamW/pg1‚2.671209786509736e-05 & lr-AdamW/pg2‚2.671209786509736e-05 trainer/global_step‚8349 % _timestamp‚1.7884135953766208e+09 _runtime‚6205.788764006 _step‚682ª—DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚223402236754944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚260814636929024 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168581813 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚234842898981888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚234842880978944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 168581813 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64063156 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 278342364 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 168601851 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64063154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109760553 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚248844478494720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246609332393984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47264810 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚260814402404352 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _runtime‚6205.789724335 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109760551 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚223402219715584 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 278362404 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚248844454554624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚234842898981888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚246609519396864 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111327966 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚235134958889936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 % _timestamp‚1.7884135953773944e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11766620414976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168601851 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚248844454554624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚234842898981888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚246609519396864 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111327966 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚235134958889936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 % _timestamp‚1.7884135953773944e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11766620414976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168601851 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚223402236754944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚260814636929024 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168581813 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚234842898981888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚234842880978944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 168581813 @8DeviceStatsMonitor.on_train_bņ²Q—atch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64063156 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 278342364 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 168601851 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64063154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109760553 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚248844478494720 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246609332393984« ü)2³ - _wandbruntime‚6206  _runtime‚6206··»÷&´ ñM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11767844553216 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚246637994896896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11768013566464 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚248873391442944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47270245 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚260844968209920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 278374810 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚248873367502848 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11768013566464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111340778 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47269695 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚234870150343680 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11734111299744 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚234870168346624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚234870168346624 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11767844553216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚223428162897920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 278394876 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109773210 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64070531 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚234870150343680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 278374810 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64070533 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11733946178132 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168621664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11971576766976 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109773212 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111340226 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 % _timestamp‚1.7884135962497246e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109773210 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚246638181913088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚260844729504256 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 278394876 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11971362001408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚552 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 168601600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚223428179937280 _runtime‚6206.662344954 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944äWDúü&µ öM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚223428179937280 _runtime‚6206.662344954 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11767844553216 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚246637994896896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11768013566464 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚248873391442944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47270245 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚260844968209920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 278374810 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚248873367502848 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11768013566464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111340778 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47269695 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚234870150343680 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11734111299744 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚234870168346624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚234870168346624 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11767844553216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚223428162897920 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 278394876 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109773210 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64070531 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚234870150343680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 278374810 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64070533 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11733946178132 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168621664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11971576766976 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109773212 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111340226 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 % _timestamp‚1.7884135962497246e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109773210 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚246638181913088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚260844729504256 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 278394876 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11971362001408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚552 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 168601600¬–e®ö2¶ - _wandbruntime‚6206  _runtime‚6206-†½§Y· Ó 0loss/FRt1/assignment_loss‚5.1600344351359775 0loss/FRt4/assignment_loss‚3.2120595133304595 1loss/FBt1/assignment_loss‚0.47629593000201564 1loss/SRqqt1/detection_loss‚2.9982885210602372 _step‚685 /loss/FRt3/detection_loss‚1.8972035269770358 /loss/FRt4/detection_loss‚2.0121357807517053 0loss/SRqqt3/detection_loss‚1.391453159057488 1loss/FBt3/assignment_loss‚0.23728766717984004 /loss/SRqqt4/detection_loss‚1.66503164768219 .loss/FRt3/assignment_loss‚2.11758542171231 /loss/FBt4/detection_loss‚2.0599572623477265 trainer/global_step‚8349 0loss/FBt4/assignment_loss‚0.4469653939438692 /loss/FBt1/detection_loss‚3.1284221589565275 epoch‚6 0loss/FRt2/assignment_loss‚3.4451854452490807 2loss/SRqqt1/assignment_loss‚2.0512776187172643 1loss/SRqqt4/assignment_loss‚1.511316352772216 .loss/FRt1/detection_loss‚2.961411471877779 -loss/FBt2/detection_loss‚3.51574178536733 /loss/FBt3/detection_loss‚0.7433250254432909 %loss/total_loss‚48.07732731270269 2loss/SRqqt2/assignment_loss‚1.4223519977745844 1loss/FBt2/assignment_loss‚0.33460988209299103 /loss/FRt2/detection_loss‚2.1268180068582296 1loss/SRqqt2/detection_loss‚1.5203915120739686 % _timestamp‚1.7884135962513368e+09 _runtime‚6206.663363633 2loss/SRqqt3/assignment_loss‚1.6421777963385744Œ"Uº^¸ Ø %loss/total_loss‚48.07732731270269 2loss/SRqqt2/assignment_loss‚1.4223519977745844 1loss/FBt2/assignment_loss‚0.33460988209299103 /loss/FRt2/detection_loss‚2.1268180068582296 1loss/SRqqt2/detection_loss‚1.5203915120739686 % _timestamp‚1.7884135962513368e+09 _runtime‚6206.663363633 2loss/SRqqt3/assignment_loss‚1.6421777963385744 0loss/FRt1/assignment_loss‚5.1600344351359775 0loss/FRt4/assignment_loss‚3.2120595133304595 1loss/FBt1/assignment_loss‚0.47629593000201564 1loss/SRqqt1/detection_loss‚2.9982885210602372 _step‚685 /loss/FRt3/detection_loss‚1.8972035269770358 /loss/FRt4/detection_loss‚2.0121357807517053 0loss/SRqqt3/detection_loss‚1.391453159057488 1loss/FBt3/assignment_loss‚0.23728766717984004 /loss/SRqqt4/detection_loss‚1.66503164768219 .loss/FRt3/assignment_loss‚2.11758542171231 /loss/FBt4/detection_loss‚2.0599572623477265 trainer/global_step‚8349 0loss/FBt4/assignment_loss‚0.4469653939438692 /loss/FBt1/detection_loss‚3.1284221589565275 epoch‚6 0loss/FRt2/assignment_loss‚3.4451854452490807 2loss/SRqqt1/assignment_loss‚2.0512776187172643 1loss/SRqqt4/assignment_loss‚1.511316352772216 .loss/FRt1/detection_loss‚2.961411471877779 -loss/FBt2/detection_loss‚3.51574178536733 /loss/FBt3/detection_loss‚0.7433250254432909­–™ò›¹ ‚(: ŸäÔ ñ›™ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.86' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.971999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÷º ‚(: ŸäÔ“üÌœ disk.rbd3.in‚0 disk.nvme0c0n1.in‚711.0625( proc.memory.availableMB‚ 331.16796875 disk.rbd3.out‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 cpu‚25.35452501683045 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚711.0625 proc.cpu.threads‚17% disk./.usageGB‚1097.0193061828613& disk.nvme0c0n1.out‚68429.029296875 disk.sda2.out‚0 disk.dm-0.in‚711.0625$ memory_percent‚97.97871112823486* proc.memory.percent‚17.076826095581055 disk.sda3.out‚ 246.984375 disk.drbd1777.out‚0 disk.rbd2.in‚0! disk.dm-0.out‚68429.029296875 disk.rbd0.out‚0" proc.memory.rssMB‚ 2797.8671875) disk./.usagePercent‚73.62188609843862 disk.rbd1.in‚0 disk.rbd1.out‚ 3.435546875 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda.out‚ 246.984375$ disk.nvme0n1.out‚68429.029296875 disk.dm-1.out‚ 246.984375 disk.rbd2.out‚143.9130859375 network.sent‚ 1.9654185e+07 network.recv‚ 2.259883e+06 :³‘» ‚(:† ®äÔÎàž™ gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.566' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5132 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àæ.¼ ‚(:’ ®äÔœ¿²› disk.nvme0c0n1.in‚711.0625 disk.sda3.in‚0 disk.sda3.out‚ 247.5546875 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1777.out‚0 disk.dm-1.out‚ 247.5546875 disk.rbd2.out‚143.9130859375' disk.nvme0c0n1.out‚68528.8525390625 disk.drbd1777.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 331.43359375% disk./.usageGB‚1097.1439590454102 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 3.435546875% disk.nvme0n1.out‚68528.8525390625* proc.memory.percent‚17.076826095581055 cpu‚25.3537643411282 network.recv‚ 2.265645e+06 disk.rbd0.in‚0" disk.dm-0.out‚68528.8525390625 disk.nvme0n1.in‚711.0625 disk.dm-0.in‚711.0625 disk.sda1.out‚0$ memory_percent‚97.97708988189697 disk.dm-1.in‚0 disk.sda.out‚ 247.5546875" proc.memory.rssMB‚ 2797.8671875 proc.cpu.threads‚17 network.sent‚ 1.9762045e+07' disk./.usagePercent‚73.630251656635Û2s£‘½ ‚(:† ½äÔ›å™ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.999' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óÇcŽ"¾ ‚(:— ½äÔõ®œ disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 248.1171875 disk.sda1.in‚0 disk.dm-1.in‚0 cpu‚25.353029882031276% disk./.usageGB‚1097.1434440612793 disk.rbd1.out‚ 3.435546875( proc.memory.availableMB‚ 330.93359375 proc.cpu.threads‚17 network.sent‚ 1.9768115e+07) disk./.usagePercent‚73.63021709561787 disk.rbd3.in‚0' disk.nvme0c0n1.out‚68627.9423828125 disk.sda1.out‚0" disk.dm-0.out‚68627.9423828125 disk.dm-1.out‚ 248.1171875 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 248.1171875 disk.drbd1777.in‚0$ memory_percent‚97.98014163970947" proc.memory.rssMB‚ 2797.8671875 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚711.0625 disk.dm-0.in‚711.0625 disk.rbd0.in‚0* proc.memory.percent‚17.076826095581055 network.recv‚ 2.270087e+06 disk.sda2.out‚0% disk.nvme0n1.out‚68627.9423828125 disk.nvme0c0n1.in‚711.0625 disk.sda.in‚0 disk.rbd0.out‚0! disk.rbd2.out‚144.07763671875ÄÞÄÃ2¿ - _wandbruntime‚6240  _runtime‚6240yGCÇÀ Á % _timestamp‚1.7884136303791835e+09 _runtime‚6240.791367223 _step‚686 & lr-AdamW/pg1‚2.607137151974694e-05 & lr-AdamW/pg2‚2.607137151974694e-05 trainer/global_step‚8399,¶i ÌÁ Æ _step‚686 & lr-AdamW/pg1‚2.607137151974694e-05 & lr-AdamW/pg2‚2.607137151974694e-05 trainer/global_step‚8399 % _timestamp‚1.7884136303791835e+09 _runtime‚6240.791367223®·3UÀ2 - _wandbruntime‚6240  _runtime‚6240…P‚ßÐ à æO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚236206349215744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64432004 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47536202 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11836291737600 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚224699395871744 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚236501407664380 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 279988056 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚262331153612800 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚236206367218688 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚262330914893824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚236206367218688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 _step‚687 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LBDeviceStatsMonitor.oí¶Tn_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111968208 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 111967681 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788413630379958e+09 _runtime‚6240.792263281 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚248042471953408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11836122737664 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169574515 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11802028831996 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 169594553 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47535677 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 279968016 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚248042658956288 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11802193953400 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110393503 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169594553 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚248042658956288 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 trainer/global_step‚8399 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110393501 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12041027706880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 279988056 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚262331153612800 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚236206367218688 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚262330914893824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚236206367218688 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 _step‚687 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111968208 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 111967681 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788413630379958e+09 _runtime‚6240.792263281 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚248042471953408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11836122737664 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169574515 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11802028831996 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 169594553 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47535677 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 279968016 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚248042658956288 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11802193953400 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110393503 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169594553 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚248042658956288 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 trainer/global_step‚8399 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110393501 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12041027706880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280000462 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64439383 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169614366 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47541116 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 _runtime‚ 6241.44778029 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11837684889088 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11803417996540 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _step‚688 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64439381 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12042230649344 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110406160 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111981025 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111980497 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚236528740011260 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 280020528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11837515875840 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚224725322014720 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚248071321472512 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚248071134456320 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280020528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110406160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚236233618580480 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚236233636583424 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169594302 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚224725339054080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11803583118152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11837684889088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚250319038854144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚250319014914048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚262361480074752 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280000462 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110406162 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110406162 trainer/global_step‚8399 % _timestamp‚1.7884136310350213e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169614366 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12042441220608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚248071321472512 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚236528922172232 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚236233636583424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47541642 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚262361245563392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 2698777603,©¨Ç õM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11803583118152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11837684889088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚250319038854144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚250319014914048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚262361480074752 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280000462 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110406162 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110406162 trainer/global_step‚8399 % _timestamp‚1.7884136310350213e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169614366 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12042441220608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚248071321472512 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚236528922172232 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚236233636583424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47541642 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚262361245563392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169594302 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280000462 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64439383 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169614366 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47541116 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 _runtime‚ 6241.44778029 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11837684889088 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11803417996540 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _step‚688 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64439381 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12042230649344 SEDeviceStatsMonitor.on_train_batch_end/requested_b]F:ZSytes.small_pool.peak‚ 406789969 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110406160 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111981025 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111980497 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚236528740011260 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 280020528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11837515875840 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚224725322014720 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚248071321472512 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚248071134456320 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280020528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110406160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚236233618580480 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚236233636583424 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169594302 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚224725339054080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199° ¶2È - _wandbruntime‚6241  _runtime‚6241¤¦ïVÉ Ð 0loss/FBt1/assignment_loss‚0.6889677444475334 .loss/FRt3/detection_loss‚2.045627052796648 0loss/SRqqt1/detection_loss‚2.576841202649203 1loss/SRqqt2/detection_loss‚1.4720233959310196 1loss/SRqqt4/detection_loss‚1.7171094775722737 0loss/FRt3/assignment_loss‚2.5027748752147594 2loss/SRqqt2/assignment_loss‚1.4035967610064237 0loss/FBt2/assignment_loss‚0.2066981546900024 ,loss/FBt4/detection_loss‚2.6221106890589 trainer/global_step‚8399 _runtime‚6241.449110676 _step‚689 /loss/FBt2/detection_loss‚3.8376730233430862 /loss/FBt3/detection_loss‚0.8906371530067645 2loss/SRqqt4/assignment_loss‚1.7053023461709942 /loss/FRt4/detection_loss‚2.0707187981021646 0loss/FRt4/assignment_loss‚3.5249039226648757 /loss/FRt1/detection_loss‚2.6416144699885926 % _timestamp‚1.7884136310367684e+09 %loss/total_loss‚49.72885143262327 /loss/FRt2/detection_loss‚2.4140520664148553 /loss/FBt1/detection_loss‚3.4726546307404837 epoch‚6 /loss/FRt2/assignment_loss‚3.357537700686344 0loss/FBt3/assignment_loss‚0.2879610534933583 0loss/FBt4/assignment_loss‚0.5709707198246861 0loss/SRqqt3/detection_loss‚1.383176985178508 /loss/FRt1/assignment_loss‚4.491674094364561 2loss/SRqqt1/assignment_loss‚2.0666603274417645 1loss/SRqqt3/assignment_loss‚1.777564787835467À¯.Þ[Ê Õ 0loss/FRt4/assignment_loss‚3.5249039226648757 /loss/FRt1/detection_loss‚2.6416144699885926 % _timestamp‚1.7884136310367684e+09 %loss/total_loss‚49.72885143262327 /loss/FRt2/detection_loss‚2.4140520664148553 /loss/FBt1/detection_loss‚3.4726546307404837 epoch‚6 /loss/FRt2/assignment_loss‚3.357537700686344 0loss/FBt3/assignment_loss‚0.2879610534933583 0loss/FBt4/assignment_loss‚0.5709707198246861 0loss/SRqqt3/detection_loss‚1.383176985178508 /loss/FRt1/assignment_loss‚4.491674094364561 2loss/SRqqt1/assignment_loss‚2.0666603274417645 1loss/SRqqt3/assignment_loss‚1.777564787835467 0loss/FBt1/assignment_loss‚0.6889677444475334 .loss/FRt3/detection_loss‚2.045627052796648 0loss/SRqqt1/detection_loss‚2.576841202649203 1loss/SRqqt2/detection_loss‚1.4720233959310196 1loss/SRqqt4/detection_loss‚1.7171094775722737 0loss/FRt3/assignment_loss‚2.5027748752147594 2loss/SRqqt2/assignment_loss‚1.4035967610064237 0loss/FBt2/assignment_loss‚0.2066981546900024 ,loss/FBt4/detection_loss‚2.6221106890589 trainer/global_step‚8399 _runtime‚6241.449110676 _step‚689 /loss/FBt2/detection_loss‚3.8376730233430862 /loss/FBt3/detection_loss‚0.8906371530067645 2loss/SRqqt4/assignment_loss‚1.7053023461709942 /loss/FRt4/detection_loss‚2.0707187981021646±¢Êã‘Ë ‚(:† ÌäÔ¡î©™ gpu.0.gpu‚21.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.917' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3834 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘.ÃÆÌ ‚(:’ ÌäÔ¯œ disk.rbd0.out‚0 disk.nvme0n1.in‚711.0625 disk.dm-0.in‚711.0625" proc.memory.rssMB‚ 2797.8671875! disk.rbd2.out‚144.07763671875 disk.sda3.out‚ 248.515625' proc.memory.availableMB‚ 331.1171875* proc.memory.percent‚17.076826095581055 cpu‚25.352307112115845 disk.rbd1.out‚ 3.435546875 disk.sda2.out‚0 disk.sda.out‚ 248.515625 disk.sda1.in‚0# memory_percent‚97.9790210723877 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 network.recv‚ 2.273917e+06 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.drbd1777.out‚0% disk.nvme0n1.out‚68727.1376953125" disk.dm-0.out‚68727.1376953125% disk./.usageGB‚1097.3934440612793 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0 network.sent‚ 1.9773467e+07 disk.dm-1.out‚ 248.515625 disk.rbd3.in‚0 disk.nvme0c0n1.in‚711.0625' disk.nvme0c0n1.out‚68727.1376953125) disk./.usagePercent‚73.64699480538185VäÒœÍ ‚(:‘ ÛäԳƾ™ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.969' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.993799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕvC Î ‚(:• ÛäÔÚÓ­œ disk.sda2.in‚0 disk.sda1.in‚0) disk./.usagePercent‚73.64699480538185% disk./.usageGB‚1097.3934440612793' disk.nvme0c0n1.out‚68826.2197265625" proc.memory.rssMB‚ 2797.8671875% disk.nvme0n1.out‚68826.2197265625 disk.dm-1.in‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 331.36328125 disk.nvme0n1.in‚711.0625 disk.dm-0.in‚711.0625 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚ 2.2796e+06 disk.dm-1.out‚ 249.1328125 disk.sda2.out‚0 disk.sda3.out‚ 249.1328125 network.sent‚ 1.9881545e+07 disk.rbd1.in‚0$ memory_percent‚97.97751903533936* proc.memory.percent‚17.076826095581055 proc.cpu.threads‚17 disk.rbd1.out‚ 3.435546875" disk.dm-0.out‚68826.2197265625 disk.sda.out‚ 249.1328125 disk.drbd1777.out‚0 cpu‚25.351584623341548 disk.rbd2.in‚0! disk.rbd2.out‚144.21826171875 disk.rbd3.out‚0 disk.nvme0c0n1.in‚711.0625…2ʵ2Ï - _wandbruntime‚6275  _runtime‚6275×¼ÉÐ Ã ' lr-AdamW/pg1‚2.5435739687498054e-05 ' lr-AdamW/pg2‚2.5435739687498054e-05 trainer/global_step‚8449 % _timestamp‚1.7884136653476105e+09 _runtime‚6275.759710858 _step‚6904MþÎÑ È _runtime‚6275.759710858 _step‚690 ' lr-AdamW/pg1‚2.5435739687498054e-05 ' lr-AdamW/pg2‚2.5435739687498054e-05 trainer/global_step‚8449 % _timestamp‚1.7884136653476105e+09²z9S2Ò - _wandbruntime‚6275  _runtime‚6275~u¦,í'Ó çO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111026451 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _step‚691 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111026451 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12111698108928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚237569835455488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11905963060224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64800856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47808669 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 112609525 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚237868220760352 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 170567217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚249475798515712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11905963060224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚237868038599588 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 170587255 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 170567217 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11871500650404 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12111910790656 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 _runtime‚6275.760639888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281613708 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111026453 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11905794060288 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚249475611512832 % _timestamp‚1.7884136653483124e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚225996537949184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64800854 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 281593668 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚237569835455488 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111026453 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚251735773317120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚249475798515712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚237569817452544 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11905794060288 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 170587255 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199«l}ò'Ô ìO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _step‚691 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111026451 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12111698108928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚237569835455488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11905963060224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64800856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47808669 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 112609525 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚237868220760352 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 170567217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚249475798515712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11905963060224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚237868038599588 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 170587255 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 170567217 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11871500650404 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12111910790656 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 _runtime‚6275.760639888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281613708 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111026453 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11905794060288 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚249475611512832 % _timestamp‚1.7884136653483124e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚225996537949184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64800854 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 281593668 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚237569835455488 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111026453 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚251735773317120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚249475798515712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚237569817452544 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11905794060288 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 170587255 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111026451³ñÇó2Õ - _wandbruntime‚6276  _runtime‚6276‰ý.äÖ ñM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170607068 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚226022498170880 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 C=DeviceStatsMon.e  itor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111039112 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚237895370946468 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _runtime‚6276.408635946 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111039112 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111039110 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112621631 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚249504274015744 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚237597086817280 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12113113882624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚237895553107440 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11907356211712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170587004 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚263878014913536 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12113328648192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚237597104820224 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281646180 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚226022481131520 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚237597104820224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚249504461031936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64808233 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47813915 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚692 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚263877776207872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281646180 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚249504461031936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚237597086817280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884136659960785e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11907187198464 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚517 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281626114 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11907356211712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 170587004 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 281626114 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170607068 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111039110 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0þ•ü&× öM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111039112 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚237895370946468 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _runtime‚6276.408635946 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111039112 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111039110 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112621631 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚249504274015744 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚237597086817280 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12113113882624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚237895553107440 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11907356211712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170587004 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚263878014913536 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12113328648192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚237597104820224 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281646180 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚226022481131520 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚237597104820224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚249504461031936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64808233 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47813915 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚692 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚263877776207872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281646180 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚249504461031936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚237597086817280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884136659960785e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11907187198464 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚517 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281626114 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11907356211712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 170587004 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 281626114 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170607068 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111039110 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170607068 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚226022498170880 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416´s3¿2Ø - _wandbruntime‚6276  _runtime‚6276d¢¨XÙ Ò /loss/FBt3/detection_loss‚1.0841685893125277 % _timestamp‚1.7884136659980373e+09 .loss/FRt1/detection_loss‚3.073458744929387 0loss/SRqqt3/detection_loss‚1.335715506541527 .loss/FBt4/detection_loss‚2.428208166360855 1loss/SRqqt2/assignment_loss‚1.612680378890432 1loss/FBt2/assignment_loss‚0.34772466868162155 /loss/FRt3/detection_loss‚1.6606502459565204 .loss/FRt4/detection_loss‚2.465838210923331 1loss/SRqqt2/detection_loss‚1.5106917141991503 /loss/FRt3/assignment_loss‚2.740276319767112 2loss/SRqqt4/assignment_loss‚1.7757979026612114 0loss/FBt4/assignment_loss‚0.7336107821211953 &loss/total_loss‚50.188399765804476 trainer/global_step‚8449 _runtime‚6276.410085176 2loss/SRqqt1/assignment_loss‚1.9005498404686267 0loss/FBt1/assignment_loss‚0.4032292558960459 .loss/FBt2/detection_loss‚4.668468475341797 epoch‚6 /loss/FRt1/assignment_loss‚4.163153231143951 /loss/FRt2/assignment_loss‚3.046746000647545 1loss/SRqqt1/detection_loss‚2.9026784793688702 .loss/FBt1/detection_loss‚2.837062492966652 _step‚693 /loss/FRt2/detection_loss‚2.4335229342634026 0loss/FRt4/assignment_loss‚3.6011472599846974 2loss/SRqqt3/assignment_loss‚1.4745855715315221 1loss/FBt3/assignment_loss‚0.14527716628451492 1loss/SRqqt4/detection_loss‚1.8431578275619769 Ž¡]Ú × .loss/FRt4/detection_loss‚2.465838210923331 1loss/SRqqt2/detection_loss‚1.5106917141991503 /loss/FRt3/assignment_loss‚2.740276319767112 2loss/SRqqt4/assignment_loss‚1.7757979026612114 0loss/FBt4/assignment_loss‚0.7336107821211953 &loss/total_loss‚50.188399765804476 trainer/global_step‚8449 _runtime‚6276.410085176 2loss/SRqqt1/assignment_loss‚1.9005498404686267 0loss/FBt1/assignment_loss‚0.4032292558960459 .loss/FBt2/detection_loss‚4.668468475341797 epoch‚6 /loss/FRt1/assignment_loss‚4.163153231143951 /loss/FRt2/assignment_loss‚3.046746000647545 1loss/SRqqt1/detection_loss‚2.9026784793688702 .loss/FBt1/detection_loss‚2.837062492966652 _step‚693 /loss/FRt2/detection_loss‚2.4335229342634026 0loss/FRt4/assignment_loss‚3.6011472599846974 2loss/SRqqt3/assignment_loss‚1.4745855715315221 1loss/FBt3/assignment_loss‚0.14527716628451492 1loss/SRqqt4/detection_loss‚1.8431578275619769 /loss/FBt3/detection_loss‚1.0841685893125277 % _timestamp‚1.7884136659980373e+09 .loss/FRt1/detection_loss‚3.073458744929387 0loss/SRqqt3/detection_loss‚1.335715506541527 .loss/FBt4/detection_loss‚2.428208166360855 1loss/SRqqt2/assignment_loss‚1.612680378890432 1loss/FBt2/assignment_loss‚0.34772466868162155 /loss/FRt3/detection_loss‚1.6606502459565204µ€+ Û ‚(:… êäÔÞ鯙 gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚95.731' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.1462 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯£þ "Ü ‚(:— êäÔ—ÚÀœ disk.dm-0.in‚711.0625! disk.rbd2.out‚145.72998046875" memory_percent‚97.977614402771 network.recv‚ 2.284536e+06( disk.nvme0c0n1.out‚68826.90576171875 disk.sda3.out‚ 249.6640625# disk.dm-0.out‚68826.90576171875" proc.memory.rssMB‚ 2797.8671875% disk./.usageGB‚1097.3934440612793 disk.drbd1777.out‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1777.in‚0& disk.nvme0n1.out‚68826.90576171875 disk.rbd0.in‚0 disk.rbd3.in‚0* proc.memory.percent‚17.076826095581055 network.sent‚ 1.9887774e+07 disk.sda.out‚ 249.6640625 disk.sda1.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚711.0625 disk.rbd3.out‚0 disk.sda.in‚0 disk.rbd1.out‚ 3.435546875( proc.memory.availableMB‚ 331.34765625 cpu‚25.35086648529334 disk.dm-1.out‚ 249.6640625 disk.rbd0.out‚0 disk.sda1.in‚0) disk./.usagePercent‚73.64699480538185 disk.nvme0c0n1.in‚711.0625 disk.sda2.in‚0 disk.rbd2.in‚0mÓlŽÝ ‚(:ƒ ùäÔÂÇæš gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚84.425' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚16.885 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·ŒÎ%Þ ‚(:š ùäÔâèã› disk.sda.out‚ 250.2578125 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.rbd0.in‚0$ memory_percent‚98.46737384796143 cpu‚25.362610199470122 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚711.0625 disk.dm-1.out‚ 250.2578125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚711.0625 disk.sda3.out‚ 250.2578125) proc.memory.percent‚17.56584644317627# disk.dm-0.out‚68925.98388671875 disk.rbd3.out‚0# proc.memory.rssMB‚ 2877.98828125 network.sent‚ 1.9996398e+07) disk./.usagePercent‚73.64699480538185! disk.rbd2.out‚145.80810546875 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.rbd1.out‚ 3.435546875 disk.dm-1.in‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚68925.98388671875 proc.cpu.threads‚17% disk./.usageGB‚1097.3934440612793 disk.sda.in‚0 disk.nvme0n1.in‚711.0625& disk.nvme0n1.out‚68925.98388671875 disk.sda2.out‚0( proc.memory.availableMB‚ 251.10546875 network.recv‚ 2.290845e+06 disk.rbd2.in‚0Üf‚{ß ‚(:„ ˆŽäԧק™ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.424' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6848 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äfwà ‚(:” ˆŽäÔÔ‘Ì› disk.sda2.out‚0 disk.rbd1.out‚ 3.435546875 disk.nvme0n1.in‚711.0625# disk.dm-0.out‚69025.97119140625$ memory_percent‚98.46725463867188 disk.sda.in‚0 disk.sda.out‚ 250.5859375 disk.drbd1777.in‚0& disk.nvme0n1.out‚69025.97119140625( disk./.usagePercent‚73.6805502249098 disk.sda3.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚69025.97119140625# proc.memory.rssMB‚ 2877.98828125 disk.rbd0.in‚0) proc.memory.percent‚17.56584644317627 proc.cpu.threads‚17 network.recv‚ 2.294653e+06% disk./.usageGB‚1097.8934440612793 disk.sda1.out‚0! disk.rbd2.out‚145.80810546875# proc.memory.availableMB‚251.125 cpu‚25.374451700774223 disk.sda2.in‚0 disk.dm-0.in‚711.0625 disk.dm-1.out‚ 250.5859375 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚711.0625 network.sent‚ 2.0001709e+07 disk.sda3.out‚ 250.5859375 disk.drbd1777.out‚0 disk.dm-1.in‚0 disk.rbd2.in‚0!´iÄ‘á ‚(:† —ŽäÔÉþ±™ gpu.0.gpu‚35.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚120.962' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1924 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0͹#ûâ ‚(:“ —ŽäÔñ†Àœ) proc.memory.percent‚17.56584644317627% disk./.usageGB‚1097.8934516906738 disk.rbd1.out‚ 3.435546875 disk.rbd3.in‚0 disk.sda3.out‚ 251.56640625) disk./.usagePercent‚73.68055073692486 disk.drbd1777.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0$ memory_percent‚98.46878051757812 disk.sda.out‚ 251.56640625 disk.drbd1777.out‚0& disk.nvme0n1.out‚69125.25537109375 disk.dm-0.in‚711.0625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚711.0625# disk.dm-0.out‚69125.25537109375 disk.rbd0.in‚0 cpu‚25.384969358185014 disk.dm-1.out‚ 251.56640625 disk.rbd2.out‚ 145.9765625# proc.memory.availableMB‚250.875 network.sent‚ 2.0007167e+07 disk.sda1.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚711.0625 disk.rbd3.out‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2877.98828125 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 2.29831e+06 disk.sda.in‚0 disk.rbd2.in‚0( disk.nvme0c0n1.out‚69125.25537109375 disk.sda2.out‚0׫±2ã - _wandbruntime‚6332  _runtime‚6332B4ñ­ä ÍT LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2460 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 171210235 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚238875982662656 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 113310137 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚217 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚239145099554332 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 140964952 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚416 $ _timestamp‚1.788413722358344e+09 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚65222634 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 5041553408 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚11942463105024 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚265249930641920 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 127308288 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚48087503 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚217 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 171226212 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚238876000665600 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 113309857 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 5041553408 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 145311232 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚65222632 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 127308288 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚11908163883548 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚250818591078912 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 171226212 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚265250062154752 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚239144958589380 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚227236935670784 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 123925592 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 5041553408 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 171210235 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 4912020992 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚199 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 145311232 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚11942590413312 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚11942590413312 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚11942463105024 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚278 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚253101105364992 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 107572736 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 282938800 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚199,ĶwX*å ÒT D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 282938800 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚199 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2460 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 171210235 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚238875982662656 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 113310137 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚217 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚239145099554332 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 140964952 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚416 $ _timestamp‚1.788413722358344e+09 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚65222634 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 5041553408 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚11942463105024 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚265249930641920 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 127308288 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚48087503 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚217 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 171226212 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚238876000665600 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 113309857 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 5041553408 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 145311232 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚65222632 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 127308288 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚11908163883548 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚250818591078912 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 171226212 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚265250062154752 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚239144958589380 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚227236935670784 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 123925592 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 5041553408 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 171210235 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 4912020992 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚199 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 145311232 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚11942590413312 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚11942590413312 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚11942463105024 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚278 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚253101105364992 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 107572736¶›9D|2æ - _wandbruntime‚6333  _runtime‚6333,hê£^)ç ØR ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚265269883343360 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚12149174412800 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚416 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 4912020992 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚281 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 127308288 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚238894865851392 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 171217631 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 140964952 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 410712064 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1006 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚65229132 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 4757245257 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚217 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 282972887 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 111739277 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚238894847848448 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚11942939549184 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚238894865851392 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚250837660089344 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚279 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 107572736âÆNOè ÝR JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 113320548 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 123925592 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚65229134 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚985 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 406789969 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 417333248 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚217 % _timestamp‚1.7884137232615924e+09 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚250837805400576 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 131512832 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚11908387564292 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚11942812240896 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚217 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚265270014856192 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚227254820185088 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚238894847848448 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚11942939549184 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 111739279 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚416 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚250837660089344 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚239163348714332 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚26 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚48091695 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 171233608 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 171233608 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 5041553408 trainer/global_step‚8462 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚250837805400576 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚199 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚227254837224448 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 171217631 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 127308288 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 113320829 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚253120708930560 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 5041553408 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 111739277 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 4624220160 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 282956908 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚238894847848448 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚11942939549184 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚238894865851392 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚250837660089344 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚279 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 107572736 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚265269883343360 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚12149174412800 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚416 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 4912020992 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚281 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 127308288 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚238894865851392 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 171217631 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 140964952 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 410712064 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1006 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚65229132 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 4757245257 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚217 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 282972887 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 111739277 B&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5355290174484253 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1740037053823471 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.49946850538253784 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13620783388614655 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6818181872367859 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3203703463077545 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5006779432296753 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 $ _timestamp‚1.788413735696442e+09 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9802438020706177 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2053723931312561 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9786579012870789 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9849937558174133 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.44403865933418274 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.49845677614212036 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 6validation_average_jet_accuracy‚0.4254547953605652 jet/accuracy_5_of_6‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08156028389930725 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.18132716417312622 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9930710196495056 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2708333432674408 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2581562101840973 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.019354838877916336 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.40810689330101013 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9906657338142395 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.49469900131225586 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3149736225605011 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2307439148426056 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12121212482452393 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.027322610840201378 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5500820875167847 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.012118922546505928 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.012471656315028667 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14006388187408447 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5070971250534058 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3698652982711792 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8487034440040588 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5048397779464722 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9649257659912109 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4376552402973175 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14336715638637543 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.35772767663002014 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.407306045293808 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9958508610725403 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.47009867429733276 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.38093093037605286 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.31684303283691406 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.22844825685024261 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12743337452411652 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9859405755996704 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17130228877067566 particle/accuracy_2_of_4‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06853782385587692 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.34613099694252014 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 jet/accuracy_2_of_7‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3320068418979645 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07245397567749023 particle/accuracy_4_of_5‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23433105647563934 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14813819527626038 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.022789115086197853 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.677909255027771 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21332423388957977 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 *jet/accuracy_2_of_3‚0.2383144497871399 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09290267527103424 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1767241358757019 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5190050601959229 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.46819886565208435 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4581987261772156 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4509093165397644 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 8 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.44097229838371277 (loss/val_total_loss‚48.1436528583654 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24105915427207947 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3802682161331177 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.05263157933950424 *jet/accuracy_1_of_6‚0.9615384340286255 jet/accuracy_6_of_6‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993895292282104 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14626584947109222 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013843392953276634 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.47327589988708496 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13755352795124054 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_5‚0.11997567862272263 particle/accuracy_1_of_3‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1131485179066658 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.563285231590271 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4433673024177551 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4035087823867798 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 'particle/f_score‚0.3929392397403717 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_5‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 _runtime‚6346.110522151 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5011024475097656 jet/accuracy_4_of_6‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3519607484340668 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.006496415939182043 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13299447298049927 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.002765753073617816 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19918453693389893 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5103845596313477 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5255951881408691 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 particle/accuracy_1_of_6‚1 particle/accuracy_4_of_7‚1 epoch‚6 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4842582643032074 ,jet/accuracy_4_of_4‚0.002213064581155777 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3876928985118866 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6455534100532532 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9855158925056458 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4285714328289032 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_2‚1 particle/accuracy_3_of_3‚1 particle/accuracy_3_of_6‚1 jet/accuracy_3_of_7‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05773809179663658 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5526315569877625 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 jet/accuracy_5_of_7‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9887880682945251 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14953483641147614 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14513500034809113 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5208333134651184 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9823246598243713 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5112287402153015 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5080088973045349 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5288118124008179 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4166666865348816 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 +particle/specificity‚0.9500082731246948 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11434554308652878 jet/accuracy_7_of_7‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8180552124977112 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 particle/accuracy_6_of_6‚1 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.511641800403595 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20245040953159332 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3518601059913635 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18934749066829681 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.37704309821128845 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9804756045341492 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.078125 *"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.022641509771347046 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1031746044754982 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 trainer/global_step‚8462 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.17777778208255768 +jet/accuracy_3_of_3‚0.02159799635410309 (particle/accuracy‚0.8501879572868347 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5208333134651184 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.47394827008247375 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4829879403114319 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.24182164669036865 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0019919080659747124 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3219697177410126 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.48089131712913513 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1526300311088562 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.997589111328125 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23361679911613464 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.36531007289886475 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.32499998807907104 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008554055355489254 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *jet/accuracy_1_of_1‚0.4715655446052551 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4812500476837158 *jet/accuracy_1_of_5‚0.9341973066329956 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_7‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5244740843772888 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49000000953674316 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 particle/accuracy_6_of_7‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5491040349006653 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1772833615541458 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9912132024765015 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08630183339118958 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.34694990515708923 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 @&Purity/*FRt0SRqqtzØvà›*3FBt/event_proportion‚0.0037822420708835125 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4829879403114319 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1790473610162735 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18176203966140747 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2943761944770813 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 jet/accuracy_6_of_7‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9920848608016968 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.49591949582099915 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3738791346549988 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 + Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1875 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4490099549293518 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 particle/accuracy_3_of_4‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5496456027030945 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2756463587284088 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.045065298676490784 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.24537040293216705 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 particle/accuracy_5_of_6‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.021130185574293137 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.022072769701480865 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9704850316047668 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184397459030151 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16599497199058533 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 +jet/accuracy_2_of_2‚0.10781938582658768 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 _step‚696 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5230599045753479 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 *jet/accuracy_1_of_4‚0.9018016457557678 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9908046126365662 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3714178502559662 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.11777778714895248 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3978354036808014 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.31943637132644653 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2541796863079071 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6818181872367859 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365078806877136 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9873238205909729 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3118055462837219 jet/accuracy_1_of_7‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15519317984580994 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5297772288322449 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06376917660236359 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14118817448616028 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5454802513122559 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4480288028717041 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.05935903638601303 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2847222089767456 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9877729415893555 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9647988080978394 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9628472924232483 particle/accuracy_2_of_5‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2340909093618393 particle/accuracy_5_of_7‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.36159515380859375 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.45720282196998596 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.030977660790085793 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.10536398738622665 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12548182904720306 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9942327737808228 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5517060160636902 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.5128205418586731 particle/accuracy_3_of_7‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5199999809265137 particle/accuracy_4_of_6‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.17767077684402466 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.185125932097435 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4650154113769531 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5094978213310242 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19539692997932434 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.49542078375816345 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.17462100088596344 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9890435934066772 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1327614188194275 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_5‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 +jet/accuracy_3_of_6‚0.17307692766189575 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4533182978630066 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5468637943267822 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1465618759393692 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.371663898229599 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05Äö’WUí Œ§ A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05773809179663658 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5526315569877625 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 jet/accuracy_5_of_7‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9887880682945251 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14953483641147614 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14513500034809113 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5208333134651184 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9823246598243713 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5112287402153015 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5080088973045349 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5288118124008179 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4166666865348816 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 +particle/specificity‚0.9500082731246948 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11434554308652878 jet/accuracy_7_of_7‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8180552124977112 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 particle/accuracy_6_of_6‚1 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.511641800403595 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20245040953159332 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3518601059913635 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18934749066829681 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.37704309821128845 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9804756045341492 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.078125 *"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.022641509771347046 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1031746044754982 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 trainer/global_step‚8462 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.17777778208255768 +jet/accuracy_3_of_3‚0.02159799635410309 (particle/accuracy‚0.8501879572868347 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5208333134651184 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.47394827008247375 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4829879403114319 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.24182164669036865 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0019919080659747124 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3219697177410126 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.48089131712913513 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1526300311088562 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.997589111328125 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23361679911613464 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.36531007289886475 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.32499998807907104 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008554055355489254 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *jet/accuracy_1_of_1‚0.4715655446052551 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4812500476837158 *jet/accuracy_1_of_5‚0.9341973066329956 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_7‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5244740843772888 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49000000953674316 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 particle/accuracy_6_of_7‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5491040349006653 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1772833615541458 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9912132024765015 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08630183339118958 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.34694990515708923 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4829879403114319 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1790473610162735 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18176203966140747 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2943761944770813 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 jet/accuracy_6_of_7‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9920848608016968 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.49591949582099915 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3738791346549988 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 + Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1875 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4490099549293518 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 particle/accuracy_3_of_4‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5496456027030945 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2756463587284088 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.045065298676490784 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.24537040293216705 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 particle/accuracy_5_of_6‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.021130185574293137 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.022072769701480865 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9704850316047668 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184397459030151 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16599497199058533 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 +jet/accuracy_2_of_2‚0.10781938582658768 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 _step‚696 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5230599045753479 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 *jet/accuracy_1_of_4‚0.9018016457557678 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9908046126365662 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3714178502559662 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.11777778714895248 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3978354036808014 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.31943637132644653 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2541796863079071 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6818181872367859 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365078806877136 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9873238205909729 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3118055462837219 jet/accuracy_1_of_7‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15519317984580994 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5297772288322449 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06376917660236359 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14118817448616028 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5454802513122559 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4480288028717041 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.05935903638601303 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2847222089767456 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9877729415893555 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9647988080978394 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9628472924232483 particle/accuracy_2_of_5‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2340909093618393 particle/accuracy_5_of_7‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.36159515380859375 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.45720282196998596 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.030977660790085793 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.10536398738622665 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12548182904720306 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9942327737808228 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5517060160636902 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.5128205418586731 particle/accuracy_3_of_7‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5199999809265137 particle/accuracy_4_of_6‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.17767077684402466 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.185125932097435 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4650154113769531 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5094978213310242 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19539692997932434 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.49542078375816345 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.17462100088596344 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9890435934066772 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1327614188194275 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_5‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 +jet/accuracy_3_of_6‚0.17307692766189575 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4533182978630066 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5468637943267822 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1465618759393692 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.371663898229599 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 +particle/sensitivity‚0.3107452690601349 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4658661484718323 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14813819527626038 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.10288065671920776 +jet/accuracy_2_of_4‚0.34395498037338257 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0044955043122172356 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.988676130771637 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.49568966031074524 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 particle/accuracy_2_of_6‚1 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9818713665008545 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3588123023509979 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5248629450798035 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07227960228919983 *jet/accuracy_1_of_2‚0.7403314113616943 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9851411581039429 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.27655163407325745 7"Purity/2FRt2SRqqt1F½“ØK<>Bt/SRqqt_purity‚0.60615074634552 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2335585504770279 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18346133828163147 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.031609196215867996 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4801977276802063 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5355290174484253 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1740037053823471 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.49946850538253784 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13620783388614655 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6818181872367859 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3203703463077545 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5006779432296753 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 $ _timestamp‚1.788413735696442e+09 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9802438020706177 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2053723931312561 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9786579012870789 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9849937558174133 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.44403865933418274 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.49845677614212036 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 6validation_average_jet_accuracy‚0.4254547953605652 jet/accuracy_5_of_6‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08156028389930725 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.18132716417312622 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9930710196495056 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2708333432674408 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2581562101840973 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.019354838877916336 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.40810689330101013 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9906657338142395 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.49469900131225586 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3149736225605011 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2307439148426056 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12121212482452393 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.027322610840201378 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5500820875167847 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.012118922546505928 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.012471656315028667 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14006388187408447 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5070971250534058 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3698652982711792 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8487034440040588 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5048397779464722 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9649257659912109 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4376552402973175 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14336715638637543 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.35772767663002014 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.407306045293808 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9958508610725403 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.47009867429733276 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.38093093037605286 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.31684303283691406 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.22844825685024261 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12743337452411652 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9859405755996704 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17130228877067566 particle/accuracy_2_of_4‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06853782385587692 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.34613099694252014 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 jet/accuracy_2_of_7‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3320068418979645 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07245397567749023 particle/accuracy_4_of_5‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23433105647563934 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14813819527626038 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.022789115086197853 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.677909255027771 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21332423388957977 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 *jet/accuracy_2_of_3‚0.2383144497871399 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09290267527103424 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1767241358757019 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5190050601959229 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.46819886565208435 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4581987261772156 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4509093165397644 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 8 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.44097229838371277 (loss/val_total_loss‚48.1436528583654 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24105915427207947 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3802682161331177 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.05263157933950424 *jet/accuracy_1_of_6‚0.9615384340286255 jet/accuracy_6_of_6‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993895292282104 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14626584947109222 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013843392953276634 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.47327589988708496 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13755352795124054 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_5‚0.11997567862272263 particle/accuracy_1_of_3‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1131485179066658 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.563285231590271 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4433673024177551 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4035087823867798 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 'particle/f_score‚0.3929392397403717 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_5‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 _runtime‚6346.110522151 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5011024475097656 jet/accuracy_4_of_6‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3519607484340668 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.006496415939182043 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13299447298049927 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.002765753073617816 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19918453693389893 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5103845596313477 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5255951881408691 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 particle/accuracy_1_of_6‚1 particle/accuracy_4_of_7‚1 epoch‚6 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4842582643032074 ,jet/accuracy_4_of_4‚0.002213064581155777 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3876928985118866 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6455534100532532 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9855158925056458 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4285714328289032 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_2‚1 particle/accuracy_3_of_3‚1 particle/accuracy_3_of_6‚1 jet/accuracy_3_of_7‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0¸AœXDœî ‚(:‘ µŽäÔÀ‰§™ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚125.156' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.031200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CVoï ‚(:Ž µŽäÔ‰éÙœ disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 711.06640625 disk.rbd3.out‚0$ disk.nvme0c0n1.out‚ 69348.3984375% disk./.usageGB‚1097.8937034606934 disk.sda1.in‚0 disk.rbd2.out‚ 146.1171875 disk.sda3.out‚ 253.15625 cpu‚25.3967851827099 network.sent‚ 2.0121793e+07 disk.dm-0.out‚ 69348.3984375 disk.dm-1.out‚ 253.15625( proc.memory.availableMB‚ 373.12890625) disk./.usagePercent‚73.68056763342211 disk.sda.out‚ 253.15625 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚97.72260189056396) proc.memory.percent‚17.57500171661377 disk.drbd1777.in‚0 disk.rbd1.out‚ 3.435546875" disk.nvme0n1.out‚ 69348.3984375# proc.memory.rssMB‚ 2879.48828125 proc.cpu.threads‚17 network.recv‚ 2.308693e+06 disk.sda.in‚0 disk.drbd1777.out‚0" disk.nvme0c0n1.in‚ 711.06640625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 711.06640625±Ê\2ð - _wandbruntime‚6373  _runtime‚6373Â(ŒëÉñ à ' lr-AdamW/pg1‚2.4805334833071843e-05 ' lr-AdamW/pg2‚2.4805334833071843e-05 trainer/global_step‚8499 % _timestamp‚1.7884137632746937e+09 _runtime‚6373.686884152 _step‚697 X˜ÌÎò È ' lr-AdamW/pg1‚2.4805334833071843e-05 ' lr-AdamW/pg2‚2.4805334833071843e-05 trainer/global_step‚8499 % _timestamp‚1.7884137632746937e+09 _runtime‚6373.686884152 _step‚697¹ŽÅM2ó - _wandbruntime‚6373  _runtime‚6373ªåüùí'ô çO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚252119270327808 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 172027804 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48337175 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172027804 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚240121792394240 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65579206 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48336662 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11963053896762 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11997477933568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11997477933568 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172047842 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112334257 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 113915866 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284382101 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112334259 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚254416421419008 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚228421511986176 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚513 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11962888775358 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚6373.687924241 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112334259 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚266621415260160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚240121810397184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚254416445359104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 284362061 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _step‚698 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚252119457330688 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11997646933504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65579204 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚240384383722174 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172047842 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252119270327808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚266621176541184 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚240384565882938 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚228421494946816Hƒ÷_èõ ìO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _step‚698 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚252119457330688 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11997646933504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65579204 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚240384383722174 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172047842 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252119270327808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚266621176541184 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚240384565882938 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚228421494946816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚252119270327808 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 172027804 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48337175 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172027804 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚240121792394240 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65579206 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48336662 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11963053896762 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11997477933568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11997477933568 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172047842 GADeviceStatsMonitor.on_train_batch_%`25 start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112334257 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 113915866 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284382101 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112334259 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚254416421419008 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚228421511986176 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚513 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11962888775358 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚6373.687924241 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112334259 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚266621415260160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚240121810397184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚254416445359104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 284362061 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192ºiNí2ö - _wandbruntime‚6374  _runtime‚6374|ûåÛ÷&÷ ñM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _runtime‚6374.338427003 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112346918 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48342476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚254445358307328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚8499 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 % _timestamp‚1.7884137639254515e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 284394507 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚266651505474048 _step‚699 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚252148119846912 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113928521 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65586583 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65586581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284414573 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172067655 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113929059 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 284414573 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112346916 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚240149079761920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚240149061758976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚252147932830720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11999040084992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11998871071744 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚240411716069054 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11964277939902 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚266651742082560 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172047591 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚240149079761920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11964443061514 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚252148119846912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172047591 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚254445334367232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112346918 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11999040084992 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11998871071744 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0… ?Ñü&ø öM >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172067655 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113929059 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 284414573 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112346916 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚240149079761920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚240149061758976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚252147932830720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11999040084992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11998871071744 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚240411716069054 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11964277939902 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚266651742082560 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172047591 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚240149079761920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11964443061514 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚252148119846912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172047591 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚254445334367232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112346918 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11999040084992 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11998871071744 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _runtime‚6374.338427003 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112346918 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48342476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚254445358307328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚8499 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 % _timestamp‚1.7884137639254515e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 284394507 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚266651505474048 _step‚699 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚252148119846912 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113928521 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65586583 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65586581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284414573»Œ»ÃV2ù - _wandbruntime‚6374  _runtime‚6374POMÅVú Ð .loss/FRt2/detection_loss‚2.476134688986672 trainer/global_step‚8499 0loss/FBt3/assignment_loss‚0.2448475309015957 .loss/FRt1/detection_loss‚3.403627461194992 2loss/SRqqt1/assignment_loss‚2.3065011089756373 2loss/SRqqt4/assignment_loss‚1.4156125936657191 2loss/SRqqt3/assignment_loss‚1.4110308223962784 1loss/FBt2/assignment_loss‚0.32121075303439284 0loss/FBt4/assignment_loss‚0.6029626496827125 .loss/FBt1/detection_loss‚5.598108522593975 .loss/FBt4/detection_loss‚1.917244098507441 _runtime‚6374.340176489 0loss/FRt4/assignment_loss‚3.1948820365799797 1loss/FBt1/assignment_loss‚0.34746374944188574 _step‚700 /loss/FBt3/detection_loss‚0.7313615953603964 $loss/total_loss‚49.5634581560534 /loss/FRt2/assignment_loss‚3.666491871409946 2loss/SRqqt2/assignment_loss‚1.2942864928266098 /loss/FRt3/detection_loss‚1.9671846067017698 0loss/SRqqt1/detection_loss‚3.194902499516805 1loss/SRqqt2/detection_loss‚1.6648759634458814 1loss/SRqqt4/detection_loss‚1.6104619637131692 /loss/FBt2/detection_loss‚2.2951325699687004 epoch‚7 .loss/FRt1/assignment_loss‚4.70525586605072 0loss/FRt3/assignment_loss‚1.6601321037779464 -loss/FRt4/detection_loss‚2.15044883357154 1loss/SRqqt3/detection_loss‚1.3832977737486363 % _timestamp‚1.7884137639280806e+09¹ Æ[û Õ .loss/FRt1/assignment_loss‚4.70525586605072 0loss/FRt3/assignment_loss‚1.6601321037779464 -loss/FRt4/detection_loss‚2.15044883357154 1loss/SRqqt3/detection_loss‚1.3832977737486363 % _timestamp‚1.7884137639280806e+09 .loss/FRt2/detection_loss‚2.476134688986672 trainer/global_step‚8499 0loss/FBt3/assignment_loss‚0.2448475309015957 .loss/FRt1/detection_loss‚3.403627461194992 2loss/SRqqt1/assignment_loss‚2.3065011089756373 2loss/SRqqt4/assignment_loss‚1.4156125936657191 2loss/SRqqt3/assignment_loss‚1.4110308223962784 1loss/FBt2/assignment_loss‚0.32121075303439284 0loss/FBt4/assignment_loss‚0.6029626496827125 .loss/FBt1/detection_loss‚5.598108522593975 .loss/FBt4/detection_loss‚1.917244098507441 _runtime‚6374.340176489 0loss/FRt4/assignment_loss‚3.1948820365799797 1loss/FBt1/assignment_loss‚0.34746374944188574 _step‚700 /loss/FBt3/detection_loss‚0.7313615953603964 $loss/total_loss‚49.5634581560534 /loss/FRt2/assignment_loss‚3.666491871409946 2loss/SRqqt2/assignment_loss‚1.2942864928266098 /loss/FRt3/detection_loss‚1.9671846067017698 0loss/SRqqt1/detection_loss‚3.194902499516805 1loss/SRqqt2/detection_loss‚1.6648759634458814 1loss/SRqqt4/detection_loss‚1.6104619637131692 /loss/FBt2/detection_loss‚2.2951325699687004 epoch‚7¼ø«7‘ü ‚(:† ÄŽäÔ³¢™ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.896' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jн`+ý ‚(:  ÄŽäÔÇê×›" disk.nvme0c0n1.in‚ 711.06640625 disk.sda3.in‚0$ memory_percent‚97.72217273712158 cpu‚25.395980770754218% disk./.usageGB‚1097.8935203552246 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 3.435546875$ disk.nvme0n1.out‚69572.783203125 disk.dm-0.in‚ 711.06640625 proc.cpu.threads‚17 network.sent‚ 2.0245466e+07 disk.nvme0n1.in‚ 711.06640625 disk.rbd3.in‚0 disk.sda1.in‚0 network.recv‚ 2.314445e+06) disk./.usagePercent‚73.68055534506047! disk.dm-0.out‚69572.783203125 disk.rbd0.in‚0& disk.nvme0c0n1.out‚69572.783203125 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 253.47265625 disk.rbd1.in‚0# proc.memory.rssMB‚ 2879.48828125 disk.dm-1.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 253.47265625 disk.rbd2.in‚0 disk.rbd2.out‚ 147.69140625 disk.sda.out‚ 253.47265625( proc.memory.availableMB‚ 373.19921875) proc.memory.percent‚17.57500171661377ËMÓ®‘þ ‚(:† ÓŽäÔÚز™ gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.098' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6196 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Qóœ&ÿ ‚(:› ÓŽäÔûç÷œ disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 711.06640625 disk.sda2.in‚0 disk.sda2.out‚0) proc.memory.percent‚17.57500171661377 disk.sda1.out‚0 disk.drbd1777.out‚0( disk./.usagePercent‚73.6805548330454( disk.nvme0c0n1.out‚69672.00048828125 disk.drbd1777.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 254.078125 disk.sda1.in‚0 disk.dm-0.in‚ 711.06640625 disk.rbd2.out‚ 147.76953125 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 254.078125$ memory_percent‚97.72219657897949# proc.memory.rssMB‚ 2879.48828125 network.sent‚ 2.0353758e+07 network.recv‚ 2.320268e+06& disk.nvme0n1.out‚69672.00048828125 disk.rbd3.out‚0 disk.sda3.in‚0 disk.sda.out‚ 254.078125# disk./.usageGB‚1097.89351272583 disk.nvme0n1.in‚ 711.06640625 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda.in‚0' proc.memory.availableMB‚ 373.1953125# disk.dm-0.out‚69672.00048828125 disk.rbd1.out‚ 3.435546875 cpu‚25.39516150995658 proc.cpu.threads‚17ÄAb1€!‚(:… âŽäÔöñ¸™ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.235' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.647 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&Ky“!‚(:› âŽäÔÝüý› disk.drbd1777.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 254.6328125 disk.nvme0n1.in‚ 711.06640625 disk.rbd0.out‚0' proc.memory.availableMB‚ 372.9453125) proc.memory.percent‚17.57500171661377) disk./.usagePercent‚73.68054971289472 disk.rbd1.in‚0$ disk.nvme0n1.out‚69672.845703125 disk.rbd3.in‚0 disk.sda.in‚0 disk.rbd1.out‚ 3.435546875 disk.dm-1.in‚0 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.drbd1777.in‚0! disk.dm-0.out‚69672.845703125 disk.rbd0.in‚0 cpu‚25.39432511531473 network.sent‚ 2.0359976e+07 disk.sda1.in‚0 disk.rbd2.out‚ 147.76953125$ memory_percent‚97.72372245788574 network.recv‚ 2.325117e+06 disk.sda2.in‚0 disk.rbd3.out‚0% disk./.usageGB‚1097.8934364318848 disk.sda3.in‚0 disk.sda.out‚ 254.6328125 disk.dm-1.out‚ 254.6328125" disk.nvme0cÜúo£–0n1.in‚ 711.06640625& disk.nvme0c0n1.out‚69672.845703125 disk.sda1.out‚0 disk.dm-0.in‚ 711.06640625# proc.memory.rssMB‚ 2879.48828125Qaa2‚!- _wandbruntime‚6408  _runtime‚64081YÙfǃ!Á _runtime‚6408.617328039 _step‚701 & lr-AdamW/pg1‚2.418028833189507e-05 & lr-AdamW/pg2‚2.418028833189507e-05 trainer/global_step‚8549 % _timestamp‚1.7884137982051613e+09N(¬žÌ„!Æ _step‚701 & lr-AdamW/pg1‚2.418028833189507e-05 & lr-AdamW/pg2‚2.418028833189507e-05 trainer/global_step‚8549 % _timestamp‚1.7884137982051613e+09 _runtime‚6408.617328039½ªÜ {2…!- _wandbruntime‚6408  _runtime‚6408æ,ý)ì'†!æO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 285987713 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12067318256128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 114554876 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48606820 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65948056 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12275840270848 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12275629686272 $ _timestamp‚1.788413798205926e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚253552596890112 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114554335 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112967207 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12067149256192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚255862068830208 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚229718654063616 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12032525715170 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚241485260631040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12032360593766 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 _runtime‚6408.618355925 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 173040544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚241485278633984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚253552409887232 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚241751014657382 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 trainer/global_step‚8549 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112967209 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12067149256192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 173020506 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 173020506 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚268137698516480 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _step‚702 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12067318256128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚253552409887232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚255862092770304 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚241485260631040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚229718671102976 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48606281 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0¯]÷ñ'‡!ëO JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚229718671102976 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48606281 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 285987713 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12067318256128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 114554876 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48606820 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65948056 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12275840270848 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12275629686272 $ _timestamp‚1.788413798205926e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚253552596890112 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114554335 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112967207 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12067149256192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚255862068830208 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚229718654063616 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12032525715170 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚241485260631040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12032360593766 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 _runtime‚6408.618355925 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 173040544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚241485278633984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚253552409887232 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚241751014657382 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 trainer/global_step‚8549 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112967209 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12067149256192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 173020506 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 173020506 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚268137698516480 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _step‚702 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12067318256128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚253552409887232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚255862092770304 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚241485260631040 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚241778529165234 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _runtime‚6409.266929882 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114567574 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 286040225 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚241778347004262 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚241512529995776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚517 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65955433 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚241512547998720 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚241512529995776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 286020159 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112979868 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12277048727552 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12033749758310 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286040225 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114567057 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173040293 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚515 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12033914879922 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 % _timestamp‚1.7884137988541365e+09 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192äGÚ'Š!öM OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112979868 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚253581072390144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12068542394368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48611626 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚253581259406336 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12068542394368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚253581259406336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 173040293 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12277259298816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚241512[“ªÕ 529995776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚517 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65955433 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚241512547998720 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚241512529995776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 286020159 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112979868 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12277048727552 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12033749758310 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286040225 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114567057 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173040293 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚515 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12033914879922 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 % _timestamp‚1.7884137988541365e+09 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65955431 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12068711407616 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229744597245952 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚241512547998720 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 173060357 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112979866 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229744614285312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48612141 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112979866 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12068711407616 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚268168265017344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚255891005718528 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚241778529165234 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _runtime‚6409.266929882 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114567574 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 286040225 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚241778347004262 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚257307716241408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12346451619328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚254985549446656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48875875 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12101997533578 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174033246 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269654404482560 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 % _timestamp‚1.7884138329082572e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚254985736449536 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚269654167860736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚706 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12136989578752 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 287613365 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174013208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚254985736449536 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113600159 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚242848746870784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48876405 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231015813180416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚254985549446656 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚242848728867840 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 115193311 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66316906 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12346664301056 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 46242201608hªi—!ìO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 287613365 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174013208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚254985736449536 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113600159 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚242848746870784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48876405 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231015813180416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚254985549446656 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚242848728867840 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 115193311 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66316906 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12346664301056 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 115192779 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚257307740181504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚231015830219776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113600159 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243117827753354 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12136820578816 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚257307716241408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12346451619328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚254985549446656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48875875 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12101997533578 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174033246 QCDeviceSta {f‰ tsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269654404482560 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 % _timestamp‚1.7884138329082572e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚254985736449536 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚269654167860736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚706 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12136989578752 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ÂwÍN2˜!- _wandbruntime‚6443  _runtime‚6443(¿éö&™!ðM bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12348079326208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48881488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66324283 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 174032995 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚231041756362752 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚504 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287665877 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚231041773402112 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12103221576718 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115205771 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12138213716992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 287645811 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚255014211949568 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113612816 % _timestamp‚1.7884138335549724e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚242876016235520 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174053059 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113612818 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚243144977939470 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48880986 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚255014398965760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113612816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66324281 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255014398965760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174032995 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚257336629189632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _step‚707 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115205267 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚257336653129728 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12138382730240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 287665877 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚269684732455936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚269684500041728 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚243145160100442 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 trainer/global_step‚8599 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174053059 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚502 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12103386698330 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 287645811 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 6443.96763402Žç¤=û&š!õM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48880986 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚255014398965760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113612816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66324281 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255014398965760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174032995 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚257336629189632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _step‚707 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115205267 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚257336653129728 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12138382730240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 287665877 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚269684732455936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚269684500041728 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚243145160100442 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 trainer/global_step‚8599 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174053059 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚502 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12103386698330 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 287645811 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 6443.96763402 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12348079326208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48881488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66324283 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 174032995 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚231041756362752 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚504 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287665877 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚231041773402112 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12103221576718 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115205771 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12138213716992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 287645811 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚255014211949568 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113612816 % _timestamp‚1.7884138335549724e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚242876016235520 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174053059 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113612818 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚243144977939470 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248ÃnT2›!- _wandbruntime‚6443  _runtime‚6443—<ÀEVœ!Ð /loss/FRt2/assignment_loss‚3.045437665212722 /loss/FRt4/assignment_loss‚2.875618553161621 2loss/SRqqt1/assignment_loss‚2.6681332878164343 1loss/FBt1/assignment_loss‚0.19833704171211744 .loss/FBt2/detection_loss‚3.170117974281311 2loss/SRqqt2/assignment_loss‚1.4110427839267585 1loss/SRqqt3/assignment_loss‚1.100655794685537 0loss/FBt4/assignment_loss‚0.5382513751780303 .loss/FRt1/detection_loss‚3.036206533908844 /loss/FBt1/detection_loss‚2.8598359326521554 epoch‚7 1loss/SRqqt4/assignment_loss‚1.235796948251399 1loss/FBt2/assignment_loss‚0.35209640327931474 1loss/FBt3/assignment_loss‚0.24793645154122684 .loss/FRt2/detection_loss‚2.566710182598659 .loss/FRt4/detection_loss‚2.055972803533077 /loss/FBt3/detection_loss‚0.6894540940598387 % _timestamp‚1.7884138335567455e+09 .loss/FRt3/detection_loss‚1.646729375720024 1loss/SRqqt1/detection_loss‚2.5200112389551625 trainer/global_step‚8599 _runtime‚6443.968786832 _step‚708 1loss/SRqqt4/detection_loss‚1.7656405784867026 /loss/FRt1/assignment_loss‚4.848033971786499 1loss/SRqqt2/detection_loss‚1.5405809685794845 .loss/FBt4/detection_loss‚2.719476123650869 %loss/total_loss‚46.87914312423942 0loss/FRt3/assignment_loss‚2.4021145021915435 0loss/SRqqt3/detection_loss‚1.384952539070086 ä3¸[!Õ 1loss/SRqqt4/assignment_loss‚1.235796948251399 1loss/FBt2/assignment_loss‚0.35209640327931474 1loss/FBt3/assignment_loss‚0.24793645154122684 .loss/FRt2/detection_loss‚2.566710182598659 .loss/FRt4/detection_loss‚2.055972803533077 /loss/FBt3/detection_loss‚0.6894540940598387 % _timestamp‚1.7884138335567455e+09 .loss/FRt3/detection_loss‚1.646729375720024 1loss/SRqqt1/detection_loss‚2.5200112389551625 trainer/global_step‚8599 _runtime‚6443.968786832 _step‚708 1loss/SRqqt4/detection_loss‚1.7656405784867026 /loss/FRt1/assignment_loss‚4.848033971786499 1loss/SRqqt2/detection_loss‚1.5405809685794845 .loss/FBt4/detection_loss‚2.719476123650869 %loss/total_loss‚46.87914312423942 0loss/FRt3/assignment_loss‚2.4021145021915435 0loss/SRqqt3/detection_loss‚1.384952539070086 /loss/FRt2/assignment_loss‚3.045437665212722 /loss/FRt4/assignment_loss‚2.875618553161621 2loss/SRqqt1/assignment_loss‚2.6681332878164343 1loss/FBt1/assignment_loss‚0.19833704171211744 .loss/FBt2/detection_loss‚3.170117974281311 2loss/SRqqt2/assignment_loss‚1.4110427839267585 1loss/SRqqt3/assignment_loss‚1.100655794685537 0loss/FBt4/assignment_loss‚0.5382513751780303 .loss/FRt1/detection_loss‚3.036206533908844 /loss/FBt1/detection_loss‚2.8598359326521554 epoch‚7Ä$uP‘ž!‚(:† äÔ—¡Å™ gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.133' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0266 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°f&&Ÿ!‚(:› äÔ®ó§œ disk.rbd0.in‚0 network.sent‚ 2.0480128e+07 disk.rbd1.out‚ 3.435546875) proc.memory.percent‚17.57500171661377 cpu‚25.39196793130381 proc.cpu.threads‚17! disk.rbd2.out‚149.54736328125% disk.nvme0c0n1.out‚69970.26171875 disk.sda3.out‚ 256.23046875 disk.rbd3.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2879.48828125 disk.sda2.out‚0 disk.sda.out‚ 256.23046875 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 711.06640625 disk.dm-0.out‚69970.26171875 disk.rbd2.in‚0 disk.drbd1777.out‚0% disk./.usageGB‚1097.8583183288574 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 711.06640625 disk.drbd1777.in‚0$ memory_percent‚97.72405624389648& proc.memory.availableMB‚ 372.890625 disk.dm-1.out‚ 256.23046875 network.recv‚ 2.33973e+06( disk./.usagePercent‚73.6781929075345# disk.nvme0n1.out‚69970.26171875 disk.dm-0.in‚ 711.06640625 disk.rbd1.in‚0 disk.sda1.out‚0P–·ð› !‚(: žäÔ«Å®š gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚89.059' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.811799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wîsé3¡!‚(:¨ žäÔȽë› network.recv‚ 2.346318e+06 disk.sda2.in‚0 cpu‚25.39114282885776) disk./.usagePercent‚73.67819393156464% disk./.usageGB‚1097.8583335876465 disk.dm-0.in‚ 711.06640625# proc.memory.rssMB‚ 2879.48828125 disk.rbd0.in‚0 disk.rbd2.in‚0 network.sent‚ 2.0589235e+07 disk.sda1.out‚0 disk.rbd1.out‚ 3.435546875 disk.nvme0n1.in‚ 711.06640625 disk.dm-1.in‚0( disk.nvme0c0n1.out‚69971.19091796875 disk.sda3.in‚0 disk.sda1.in‚0& disk.nvme0n1.out‚69971.19091796875 disk.rbd3.in‚0 disk.sda3.out‚ 256.66796875 disk.sda.out‚ 256.66796875 disk.drbd1777.out‚0" disk.nvme0c0n1.in‚ 711.06640625 disk.drbd1777.in‚0# disk.dm-0.out‚69971.19091796875 disk.dm-1.out‚ 256.66796875 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda.in‚0 disk.rbd1.in‚0) proc.memory.percent‚17.57500171661377! disk.rbd2.out‚149.54736328125 disk.rbd3.out‚0$ memory_percent‚97.72331714630127( proc.memory.availableMB‚ 373.01171875u³É­2¢!- _wandbruntime‚6478  _runtime‚6478&”48È£! trainer/global_step‚8649 $ _timestamp‚1.788413867767144e+09 _runtime‚6478.179272894 _step‚709 ' lr-AdamW/pg1‚2.2946790280487376e-05 ' lr-AdamW/pg2‚2.2946790280487376e-05q€Üͤ!Ç ' lr-AdamW/pg1‚2.2946790280487376e-05 ' lr-AdamW/pg2‚2.2946790280487376e-05 trainer/global_step‚8649 $ _timestamp‚1.788413867767144e+09 _runtime‚6478.179272894 _step‚709ÅŽr£·2¥!- _wandbruntime‚6478  _runtime‚6478:—!çO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12206491901440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚244212215107584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚256418689006080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 % _timestamp‚1.7884138677678645e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114233109 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12206660901376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 289239017 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175005910 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12417567516160 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚244484458688562 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚256418876008960 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66685754 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12206660901376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 trainer/global_step‚8649 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 175005910 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0½Þ*ò'§!ìO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 289239017 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175005910 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12417567516160 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚244484458688562 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚256418876008960 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66685754 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12206660901376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 trainer/global_step‚8649 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 175005910 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12206491901440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚244212215107584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚256418689006080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 % _timestamp‚1.7884138677678645e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114233109 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12206660901376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26Æ †ïÉ2¨!- _wandbruntime‚6478  _runtime‚6478ˆ´=/÷&©!ñM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12418988978176 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114245766 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚256447351508992 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12207885039616 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289271463 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66693133 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚271201289519104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚271201050813440 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12208054052864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚258782300540928 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 % _timestamp‚1.7884138684263449e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚258782276600832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12418774212608 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12208054052864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 289271463 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚6478.839084896 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 289291529 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114245768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49152334 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49152863 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115845465 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚244511608874678 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 _step‚711 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚232338932518912 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289291529 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚244239484472320 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚244239484472320 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12172693395126 trainer/global_step‚8649 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚244239466469376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚244511791035650 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 175045761 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175025697 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114245768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115845996 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12207885039616 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚256447538525184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66693131 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚232338915479552 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114245766 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0‹â”®Xª!öM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚271201289519104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚271201050813440 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12208054052864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚258782300540928 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 % _timestamp‚1.7884138684263449e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚258782276600832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12418774212608 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12208054052864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 289271463 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚6478.839084896 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 289291529 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114245768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49152334 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49152863 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115845465 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚244511608874678 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 _step‚711 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚232338932518912 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289291529 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚244239484472320 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚244239484472320 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMÝ>®¤onitor.on_train_batch_end/segment.small_pool.current‚199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12172693395126 trainer/global_step‚8649 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚244239466469376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚244511791035650 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 175045761 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175025697 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114245768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115845996 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12207885039616 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚256447538525184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66693131 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚232338915479552 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114245766 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12418988978176 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114245766 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚256447351508992 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12207885039616 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289271463 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66693133 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360Çxñvƒ2«!- _wandbruntime‚6478  _runtime‚6478¨%NŒW¬!Ñ trainer/global_step‚8649 /loss/FRt4/detection_loss‚2.2181253777999506 1loss/SRqqt4/detection_loss‚1.9286607351957583 $loss/total_loss‚50.1284168991719 _runtime‚6478.840462261 1loss/SRqqt3/detection_loss‚1.3605263049581222 2loss/SRqqt2/assignment_loss‚1.5229984286331362 2loss/SRqqt4/assignment_loss‚1.5023824118223845 /loss/FRt1/detection_loss‚2.9035628490588246 .loss/FBt2/detection_loss‚3.686461532115936 epoch‚7 /loss/FRt3/detection_loss‚2.0484657328398455 0loss/SRqqt1/detection_loss‚2.682734090834856 /loss/FBt1/detection_loss‚3.0322925610975786 1loss/SRqqt1/assignment_loss‚2.312507591675967 0loss/FBt1/assignment_loss‚0.6074610722022374 0loss/FBt4/assignment_loss‚0.5975073336407232 /loss/FRt2/detection_loss‚1.9621794755602706 /loss/FBt3/detection_loss‚0.9461564705072517 % _timestamp‚1.7884138684284244e+09 -loss/FRt1/assignment_loss‚5.1566461780492 /loss/FRt4/assignment_loss‚3.788674767110862 2loss/SRqqt3/assignment_loss‚1.6528482950691665 1loss/FBt2/assignment_loss‚0.14048413992859424 1loss/SRqqt2/detection_loss‚1.4681701959381181 0loss/FRt2/assignment_loss‚3.0630178718731322 0loss/FBt3/assignment_loss‚0.2539300937960683 _step‚712 /loss/FRt3/assignment_loss‚2.660715243646077 /loss/FBt4/detection_loss‚2.63190814581784335"#ž\­!Ö /loss/FRt4/detection_loss‚2.2181253777999506 1loss/SRqqt4/detection_loss‚1.9286607351957583 $loss/total_loss‚50.1284168991719 _runtime‚6478.840462261 1loss/SRqqt3/detection_loss‚1.3605263049581222 2loss/SRqqt2/assignment_loss‚1.5229984286331362 2loss/SRqqt4/assignment_loss‚1.5023824118223845 /loss/FRt1/detection_loss‚2.9035628490588246 .loss/FBt2/detection_loss‚3.686461532115936 epoch‚7 /loss/FRt3/detection_loss‚2.0484657328398455 0loss/SRqqt1/detection_loss‚2.682734090834856 /loss/FBt1/detection_loss‚3.0322925610975786 1loss/SRqqt1/assignment_loss‚2.312507591675967 0loss/FBt1/assignment_loss‚0.6074610722022374 0loss/FBt4/assignment_loss‚0.5975073336407232 /loss/FRt2/detection_loss‚1.9621794755602706 /loss/FBt3/detection_loss‚0.9461564705072517 % _timestamp‚1.7884138684284244e+09 -loss/FRt1/assignment_loss‚5.1566461780492 /loss/FRt4/assignment_loss‚3.788674767110862 2loss/SRqqt3/assignment_loss‚1.6528482950691665 1loss/FBt2/assignment_loss‚0.14048413992859424 1loss/SRqqt2/detection_loss‚1.4681701959381181 0loss/FRt2/assignment_loss‚3.0630178718731322 0loss/FBt3/assignment_loss‚0.2539300937960683 _step‚712 /loss/FRt3/assignment_loss‚2.660715243646077 /loss/FBt4/detection_loss‚2.6319081458178433 trainer/global_step‚8649È>˜[Ûœ®!‚(:‘ ­äÔñǤ™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.594' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.918799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®W*+¯!‚(:  ­äÔ²øœ disk.rbd1.out‚ 3.435546875% disk.nvme0c0n1.out‚70070.43359375 network.sent‚ 2.0595306e+07 disk.sda3.in‚0 disk.drbd1777.in‚0 cpu‚25.390353762526317 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 257.13671875 disk.nvme0n1.in‚ 711.06640625 disk.dm-0.in‚ 711.06640625( proc.memory.availableMB‚ 372.78515625 disk.rbd3.out‚0 disk.sda1.in‚0$ memory_percent‚97.72469997406006# proc.memory.rssMB‚ 2879.48828125 network.recv‚ 2.350764e+06% disk./.usageGB‚1097.8583030700684 disk.dm-1.out‚ 257.13671875 disk.rbd2.in‚0! disk.rbd2.out‚149.62548828125 disk.sda2.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚73.67819188350437 disk.rbd3.in‚0 disk.drbd1777.out‚0) proc.memory.percent‚17.57500171661377" disk.nvme0c0n1.in‚ 711.06640625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0# disk.nvme0n1.out‚70070.43359375 disk.dm-0.out‚70070.43359375 disk.dm-1.in‚0 disk.sda3.out‚ 257.13671875A”^@°!‚(:… ¼äÔÿ¶˜™ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚89.199' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.8398 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pV|ä.±!‚(:£ ¼äÔ¬Û› network.sent‚ 2.0703525e+07$ disk./.usageGB‚1097.859977722168 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda1.out‚0" disk.dm-0.out‚70173.8232421875 disk.rbd3.out‚0) proc.memory.percent‚17.57500171661377 cpu‚25.389574137831904 disk.sda3.out‚ 257.69921875 disk.sda.out‚ 257.69921875 disk.dm-1.out‚ 257.69921875 disk.sda2.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 3.435546875 network.recv‚ 2.356649e+06( disk./.usagePercent‚73.6783042708119 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚70173.8232421875 disk.rbd0.out‚0 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 711.06640625' disk.nvme0c0n1.out‚70173.8232421875( proc.memory.availableMB‚ 249.38671875 disk.drbd1777.in‚0 disk.sda3.in‚0 disk.dm-0.in‚ 711.06640625! disk.rbd2.out‚149.62548828125# memory_percent‚98.4778642654419 disk.rbd1.in‚0 disk.nvme0n1.in‚ 711.06640625 disk.sda2.out‚0# proc.memory.rssMB‚ 2879.48828125^wL9œ²!‚(:‘ ËäÔ”»•™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.372' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.674400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lg7µ+³!‚(:  ËäÔøîØ›" disk.nvme0c0n1.in‚ 711.06640625" disk.dm-0.out‚70174.1005859375 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda.out‚ 258.1796875 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 258.1796875 disk.rbd0.in‚0' disk.nvme0c0n1.out‚70174.1005859375 disk.sda3.out‚ 258.1796875 disk.drbd1777.out‚0) proc.memory.percent‚17.57500171661377( disk./.usagePercent‚73.6783058068571 proc.cpu.threads‚17% disk./.usageGB‚1097.8600006103516 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2879.48828125 network.recv‚ 2.360543e+06 disk.sda.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 3.435546875% disk.nvme0n1.out‚70174.1005859375( proc.memory.availableMB‚ 249.13671875 cpu‚25.38879480159827 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 711.06640625$ memory_percent‚98.47939014434814 network.sent‚ 2.0709031e+07 disk.dm-0.in‚ 711.06640625 disk.rbd2.in‚0 disk.rbd2.out‚149.7900390625wd ¡2´!- _wandbruntime‚6512  _runtime‚6512Ãá8ɵ!à ' lr-AdamW/pg1‚2.2338595789401577e-05 ' lr-AdamW/pg2‚2.2338595789401577e-05 trainer/global_step‚8699 % _timestamp‚1.7884139023817616e+09 _runtime‚6512.793900476 _step‚713ÀäD³Î¶!È ' lr-AdamW/pg1‚2.2338595789401577e-05 ' lr-AdamW/pg2‚2.2338595789401577e-05 trainer/global_step‚8699 % _timestamp‚1.7884139023817616e+09 _runtime‚6512.793900476 _step‚713Éë2·!- _wandbruntime‚6512  _runtime‚6512&ì ì'¸!æO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 290884709 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176018650 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12276332224000 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49418420 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12240941170394 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 trainer/global_step‚8699 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12276163224064 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚257852015568384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚257851828565504 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114866057 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚512 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚245575683344384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚510 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚260199035003904 _runtime‚6512.794909175 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12488300209152 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚272687311272960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 290864669 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176018650 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚233610131414016 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12240776048990 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚245575665341440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12276163224064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚245575683344384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49417910 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚245850907463006 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272687543700480 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚260199011063808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 116472514 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚245575665341440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚257852015568384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116473026 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12488508696576 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚257851828565504 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114866059 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67054606 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚272687311272960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 290864669 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176018650 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚233610131414016 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12240776048990 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚245575665341440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12276163224064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚245575683344384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49417910 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚245850907463006 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272687543700480 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚260199011063808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 116472514 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚245575665341440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚257852015568384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116473026 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12488508696576 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚257851828565504 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114866059 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67054606 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12242330335146 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 % _timestamp‚1.7884139030300758e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12277556362240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚260227947952128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚6513.442706335 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚245602934706176 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 290917181 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚245602952709120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12277556362240 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114878718 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚272717636371968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚245602952709120 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚245602934706176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚257880678084608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚513 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 290897115 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176038463 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114878716 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚272717877174784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 176018399 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚257880491068416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚257880678084608 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176038463 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 176018399 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49424080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚245878239809886 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114878718 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12489929222656 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12489712359936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12277725375488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114878716 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116485548 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49423567 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 116486063 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67061981 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚245878421970858 _step‚715 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 290897115 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚515 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚260227924012032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚233636091635712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12277725375488 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 290917181 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116485548 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49423567 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 116486063 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67061981 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚245878421970858 _step‚715 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 290897115 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚515 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚260227924012032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚233636091635712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12277725375488 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 290917181 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12242330335146 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 % _timestamp‚1.7884139030300758e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12277556362240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚260227947952128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚6513.442706335 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚245602934706176 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 290917181 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚245602952709120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12277556362240 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114878718 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚272717636371968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚245602952709120 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚245602934706176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚257880678084608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚513 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 290897115 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176038463 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114878716 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚272717877174784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 176018399 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚257880491068416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚257880678084608 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176038463 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 176018399 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49424080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚245878239809886 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199˪-Ö2½!- _wandbruntime‚6513  _runtime‚6513YxÊí\¾!Ö 2loss/SRqqt4/assignment_loss‚1.6642198812435656 /loss/FRt4/detection_loss‚2.1702353670483667 .loss/FBt2/detection_loss‚3.705797702074051 epoch‚7 1loss/SRqqt1/detection_loss‚2.7576906621456145 1loss/SRqqt3/detection_loss‚1.5116342515276189 _runtime‚6513.444119226 trainer/global_step‚8699 0loss/FRt2/assignment_loss‚2.3991567201392594 /loss/FRt3/assignment_loss‚2.124130075507694 1loss/SRqqt1/assignment_loss‚2.168310359120369 1loss/FBt1/assignment_loss‚0.32996826562209336 1loss/FBt4/assignment_loss‚0.42533800247583714 /loss/FRt1/assignment_loss‚5.177367108709672 /loss/FRt4/assignment_loss‚3.599147697289785 2loss/SRqqt3/assignment_loss‚1.3485688405732315 /loss/FRt2/detection_loss‚2.4467739899491154 /loss/FBt1/detection_loss‚2.5338492563792636 _step‚716 1loss/FBt2/assignment_loss‚0.02232024534714583 .loss/FRt1/detection_loss‚2.748443568454069 .loss/FRt3/detection_loss‚2.032399484432406 1loss/SRqqt4/detection_loss‚1.8987508638232362 % _timestamp‚1.7884139030317142e+09 1loss/FBt3/assignment_loss‚0.13617917284884523 /loss/FBt3/detection_loss‚0.8562647298656465 &loss/total_loss‚48.833367910692225 2loss/SRqqt2/assignment_loss‚1.3862200093745896 1loss/SRqqt2/detection_loss‚1.4395561197458149 .loss/FBt4/detection_loss‚3.951045536994934Í<°-a¿!Û epoch‚7 1loss/SRqqt1/detection_loss‚2.7576906621456145 1loss/SRqqt3/detection_loss‚1.5116342515276189 _runtime‚6513.444119226 trainer/global_step‚8699 0loss/FRt2/assignment_loss‚2.3991567201392594 /loss/FRt3/assignment_loss‚2.124130075507694 1loss/SRqqt1/assignment_loss‚2.168310359120369 1loss/FBt1/assignment_loss‚0.32996826562209336 1loss/FBt4/assignment_loss‚0.42533800247583714 /loss/FRt1/assignment_loss‚5.177367108709672 /loss/FRt4/assignment_loss‚3.599147697289785 2loss/SRqqt3/assignment_loss‚1.3485688405732315 /loss/FRt2/detection_loss‚2.4467739899491154 /loss/FBt1/detection_loss‚2.5338492563792636 _step‚716 1loss/FBt2/assignment_loss‚0.02232024534714583 .loss/FRt1/detection_loss‚2.748443568454069 .loss/FRt3/detection_loss‚2.032399484432406 1loss/SRqqt4/detection_loss‚1.8987508638232362 % _timestamp‚1.7884139030317142e+09 1loss/FBt3/assignment_loss‚0.13617917284884523 /loss/FBt3/detection_loss‚0.8562647298656465 &loss/total_loss‚48.833367910692225 2loss/SRqqt2/assignment_loss‚1.3862200093745896 1loss/SRqqt2/detection_loss‚1.4395561197458149 .loss/FBt4/detection_loss‚3.951045536994934 2loss/SRqqt4/assignment_loss‚1.6642198812435656 /loss/FRt4/detection_loss‚2.1702353670483667 .loss/FBt2/detection_loss‚3.705797702074051Ì¥h>ÞœÀ!‚(:‘ ÚäÔ“£˜™ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.436' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.887200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eÕ…+#Á!‚(:˜ ÚäÔþ¬Çœ disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 258.65625 disk.rbd1.in‚0 disk.nvme0n1.in‚ 711.06640625 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda.in‚0% disk.nvme0n1.out‚70273.4873046875" disk.nvme0c0n1.in‚ 711.06640625" disk.dm-0.out‚70273.4873046875 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd2.out‚149.7900390625$ memory_percent‚98.47662448883057( proc.memory.availableMB‚ 249.58984375# proc.memory.rssMB‚ 2879.48828125 network.recv‚ 2.365507e+06 disk.rbd0.in‚0 disk.sda.out‚ 258.65625 disk.dm-0.in‚ 711.06640625 disk.dm-1.out‚ 258.65625) proc.memory.percent‚17.57500171661377' disk.nvme0c0n1.out‚70273.4873046875 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 3.435546875 cpu‚25.3879738397484 disk.sda2.out‚0 network.sent‚ 2.0715249e+07) disk./.usagePercent‚73.67829377450299# disk./.usageGB‚1097.85982131958 disk.rbd0.out‚0 disk.rbd3.out‚0„ÿ¸ŒœÂ!‚(:‘ éäÔζž™ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.933' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.186600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09èL*Ã!‚(:Ÿ éäÔ•Ë—œ disk.sda3.in‚0 disk.sda3.out‚ 259.078125 disk.dm-1.in‚0 disk.sda.in‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 711.06640625' disk.nvme0c0n1.out‚70372.6884765625 disk.sda2.out‚0 disk.drbd1777.in‚0 network.sent‚ 2.0823988e+07 disk.sda.out‚ 259.078125 disk.nvme0n1.in‚ 711.06640625% disk.nvme0n1.out‚70372.6884765625) proc.memory.percent‚17.57500171661377) disk./.usagePercent‚73.67830759890984 disk.rbd1.in‚0 disk.rbd1.out‚ 3.435546875 disk.dm-0.in‚ 711.06640625 disk.rbd2.in‚0 cpu‚25.387204458591647 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.dm-1.out‚ 259.078125 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚98.48268032073975( proc.memory.availableMB‚ 248.59765625 network.recv‚ 2.371712e+06" disk.dm-0.out‚70372.6884765625# proc.memory.rssMB‚ 2879.48828125 disk.sda1.out‚0 disk.rbd2.out‚151.5048828125 disk.rbd3.out‚0 disk.sda2.in‚0% disk./.usageGB‚1097.8600273132324v*`w2Ä!- _wandbruntime‚6547  _runtime‚6547ÕgiÉÅ!à ' lr-AdamW/pg1‚2.1736273716284677e-05 ' lr-AdamW/pg2‚2.1736273716284677e-05 trainer/global_step‚8749 % _timestamp‚1.7884139373353775e+09 _runtime‚6547.747518928 _step‚717vÅ;YÎÆ!È ' lr-AdamW/pg1‚2.1736273716284677e-05 ' lr-AdamW/pg2‚2.1736273716284677e-05 trainer/global_step‚8749 % _timestamp‚1.7884139373353775e+09 _runtime‚6547.747518928 _step‚717Í]ù=2Ç!- _wandbruntime‚6547  _runtime‚6547cöœdßÈ!æO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚274204085944832 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259285155127808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12559403529728 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49689443 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚247217538398214 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 trainer/global_step‚8749 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚234907307570176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚274203845128704 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \IDeviceStatsMonitor.on_traM*øû in_batch_start/active_bytes.small_pool.allocated‚12346003546624 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12345834546688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177011352 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚246939151581184 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259284968124928 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115499009 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884139373363132e+09 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117112897 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26À½†ñ'É!ëO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12346003546624 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12345834546688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177011352 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚246939151581184 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259284968124928 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115499009 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884139373363132e+09 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117112897 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚274204085944832 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259285155127808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12559403529728 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49689443 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚247217538398214 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 trainer/global_step‚8749 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚234907307570176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚274203845128704 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2Î0Ä•¯2Ê!- _wandbruntime‚6548  _runtime‚6548̵—ö&Ë!ðM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚246966420945920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292542833 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 177031165 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚274234410460672 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12311637031942 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 trainer/global_step‚8749 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 292542833 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚259313817644032 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115511668 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117125606 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12560617108992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 177031165 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12347227684864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221928448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚261673571423232 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚719 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚274234188532224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115511666 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12347227684864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚259313817644032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 197988352 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚247244870745094 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12560815097344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49694775 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115511668 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚246966402942976 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408(öŠ·$Ì!õM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚246966420945920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292542833 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 177031165 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚274234410460672 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12311637031942 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 trainer/global_step‚8749 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 292542833 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚259313817644032 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115511668 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117125606 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12560617108992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 177031165 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12347227684864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221928448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚261673571423232 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚719 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚274234188532224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115511666 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12347227684864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚259313817644032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 197988352 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚247244870745094 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12560815097344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49694775 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115511668 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚246966402942976 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217ÏËÿµ2Í!- _wandbruntime‚6548  _runtime‚6548Þ†UVÎ!Ð 0loss/FRt2/assignment_loss‚3.2354437138171908 1loss/SRqqt3/assignment_loss‚1.448821550834987 /loss/SRqqt1/detection_loss‚2.54828493640973 0loss/SRqqt2/detection_loss‚1.651161518234473 /loss/FBt2/detection_loss‚2.8907452395984103 1loss/SRqqt2/assignment_loss‚1.444411904565417 0loss/FBt2/assignment_loss‚0.3860159305955416 .loss/FRt1/detection_loss‚2.795893393244062 %loss/total_loss‚47.67079746955124 trainer/global_step‚8749 2loss/SRqqt1/assignment_loss‚2.4136849756424246 2loss/SRqqt4/assignment_loss‚1.6697505563497543 0loss/SRqqt4/detection_loss‚1.976019308830683 0loss/FBt3/assignment_loss‚0.2864423440580746 0loss/FBt4/assignment_loss‚0.6353911191987814 /loss/FBt3/detection_loss‚0.7174413363387188 /loss/FRt1/assignment_loss‚4.160179703576224 .loss/FBt1/detection_loss‚3.044230369004336 0loss/FRt3/assignment_loss‚1.7809011885460386 /loss/FRt3/detection_loss‚1.9146824591654412 _runtime‚6548.398791639 _step‚720 epoch‚7 % _timestamp‚1.7884139379867396e+09 0loss/FBt1/assignment_loss‚0.4855309344496742 /loss/FRt2/detection_loss‚2.2729639664609382 /loss/FRt4/detection_loss‚2.3226366025209426 1loss/SRqqt3/detection_loss‚1.2517122500023599 /loss/FBt4/detection_loss‚2.5132302790880203 0loss/FRt4/assignment_loss‚3.8252218890190126áYgº[Ï!Õ /loss/FRt1/assignment_loss‚4.160179703576224 .loss/FBt1/detection_loss‚3.044230369004336 0loss/FRt3/assignment_loss‚1.7809011885460386 /loss/FRt3/detection_loss‚1.9146824591654412 _runtime‚6548.398791639 _step‚720 epoch‚7 % _timestamp‚1.7884139379867396e+09 0loss/FBt1/assignment_loss‚0.4855309344496742 /loss/FRt2/detection_loss‚2.2729639664609382 /loss/FRt4/detection_loss‚2.3226366025209426 1loss/SRqqt3/detection_loss‚1.2517122500023599 /loss/FBt4/detection_loss‚2.5132302790880203 0loss/FRt4/assignment_loss‚3.8252218890190126 0loss/FRt2/assignment_loss‚3.2354437138171908 1loss/SRqqt3/assignment_loss‚1.448821550834987 /loss/SRqqt1/detection_loss‚2.54828493640973 0loss/SRqqt2/detection_loss‚1.651161518234473 /loss/FBt2/detection_loss‚2.8907452395984103 1loss/SRqqt2/assignment_loss‚1.444411904565417 0loss/FBt2/assignment_loss‚0.3860159305955416 .loss/FRt1/detection_loss‚2.795893393244062 %loss/total_loss‚47.67079746955124 trainer/global_step‚8749 2loss/SRqqt1/assignment_loss‚2.4136849756424246 2loss/SRqqt4/assignment_loss‚1.6697505563497543 0loss/SRqqt4/detection_loss‚1.976019308830683 0loss/FBt3/assignment_loss‚0.2864423440580746 0loss/FBt4/assignment_loss‚0.6353911191987814 /loss/FBt3/detection_loss‚0.7174413363387188Ðø–oX‘Ð!‚(:† øäÔ˜¹´™ gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.557' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1114 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èžñÑ!‚(: øäÔ®ƒ±œ cpu‚25.386434113672966 disk.nvme0n1.out‚ 70472.3125 network.sent‚ 2.0828638e+07 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 711.06640625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.dm-0.in‚ 711.06640625% disk./.usageGB‚1097.8600883483887 disk.dm-1.out‚ 259.49609375 disk.sda1.in‚0) proc.memory.percent‚17.57500171661377 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚151.5048828125 disk.sda1.out‚0 proc.cpu.threads‚17 network.recv‚ 2.375105e+06) disk./.usagePercent‚73.67831169503039 disk.sda3.in‚0 disk.nvme0n1.in‚ 711.06640625 disk.rbd1.out‚ 3.435546875 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2879.48828125! disk.nvme0c0n1.out‚ 70472.3125 disk.dm-0.out‚ 70472.3125 disk.sda3.out‚ 259.49609375 disk.sda.out‚ 259.49609375 disk.rbd1.in‚0$ memory_percent‚98.47984313964844$ proc.memory.availableMB‚249.0625p"#¨›Ò!‚(: ‡äÔßê®™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚94.659' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.931800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ›˜g"Ó!‚(:— ‡äÔœèùœ$ memory_percent‚98.47681522369385 disk.rbd2.in‚0! disk.rbd2.out‚151.60302734375 disk.sda1.in‚0# proc.memory.rssMB‚ 2879.48828125 proc.cpu.threads‚17 network.sent‚ 2.0936646e+07% disk./.usageGB‚1097.8602104187012 disk.rbd1.out‚ 3.435546875 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 711.06640625 disk.rbd0.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 260.11328125 disk.rbd0.in‚0) disk./.usagePercent‚73.67831988727148 disk.sda3.in‚0( proc.memory.availableMB‚ 249.55859375 disk.dm-0.in‚ 711.06640625 disk.sda2.in‚0 disk.rbd1.in‚0 disk.dm-0.out‚ 70571.34375" disk.nvme0c0n1.out‚ 70571.34375 disk.sda2.out‚0 disk.sda.out‚ 260.11328125 disk.drbd1777.out‚0 cpu‚25.385702312929293 disk.dm-1.in‚0) proc.memory.percent‚17.57500171661377 network.recv‚ 2.380477e+06 disk.nvme0n1.out‚ 70571.34375 disk.rbd3.in‚0 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 711.06640625 disk.sda3.out‚ 260.11328125 disk.sda1.out‚0Z/2Ô!- _wandbruntime‚6582  _runtime‚65826`ñJÉÕ!à _runtime‚6582.553910474 _step‚721 ' lr-AdamW/pg1‚2.1139949584153953e-05 ' lr-AdamW/pg2‚2.1139949584153953e-05 trainer/global_step‚8799 % _timestamp‚1.7884139721417708e+09SV´%ÎÖ!È ' lr-AdamW/pg1‚2.1139949584153953e-05 ' lr-AdamW/pg2‚2.1139949584153953e-05 trainer/global_step‚8799 % _timestamp‚1.7884139721417708e+09 _runtime‚6582.553910474 _step‚721Ñ.çe2×!- _wandbruntime‚6582  _runtime‚6582;pë'Ø!åO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12415674869248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚263090329826304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚263090305886208 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116131959 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12630253793280 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116131957 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 294136013 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚248302601815040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 177984016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚275720346997760 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12379719685806 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚8799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12415674869248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67792304 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177984016 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49961067 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚248584169333422 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚260718107684352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚260718294687232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49960537 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49961067 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚248584169333422 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚260718107684352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚260718294687232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49960537 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12415674869248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚263090329826304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚263090305886208 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116131959 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12630253793280 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116131957 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 294136013 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚248302601815040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 177984016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚275720346997760 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12379719685806 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚8799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12415674869248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67792304 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177984016 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚248329871179776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116144618 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49966830 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 % _timestamp‚1.7884139727883298e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚248329871179776 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚248329889182720 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 117766513 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294168485 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117766008 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚248611683841274 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12417068020736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚248611501680302 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚260746770187264 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚248329889182720 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116144616 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚236230392829952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178003803 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49966327 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12416899007488 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚275750676454912 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116144618 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116144616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚263119218834432 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12381273971962 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚723Ù†ü&Ü!öM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12416899007488 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚275750676454912 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116144618 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116144616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚263119218834432 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12381273971962 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚723 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178023867 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚260746957203456 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚263119242774528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67799683 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67799681 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12381108850350 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚260746957203456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚248329871179776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116144618 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49966830 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 % _timestamp‚1.7884139727883298e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚248329871179776 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚248329889182720 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 117766513 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294168485 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117766008 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚248611683841274 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12417068020736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚248611501680302 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚260746770187264 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚248329889182720 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116144616 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚236230392829952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178003803 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49966327 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992Ó‡Ù^2Ý!- _wandbruntime‚6583  _runtime‚6583Oëe¤WÞ!Ñ 1loss/SRqqt1/detection_loss‚2.5567270506705557 _step‚724 0loss/FRt2/assignment_loss‚3.2100103933077593 0loss/FBt4/assignment_loss‚0.6360510086566137 /loss/FRt4/detection_loss‚2.0989039917786916 /loss/FBt1/detection_loss‚2.4282903000712395 trainer/global_step‚8799 0loss/FRt3/assignment_loss‚2.5258544042425335 1loss/SRqqt4/assignment_loss‚1.499025277936688 .loss/FRt1/detection_loss‚2.919695858160655 .loss/FBt2/detection_loss‚3.462868893146515 .loss/FBt3/detection_loss‚0.751189925198206 epoch‚7 &loss/total_loss‚47.859347496636325 1loss/SRqqt3/assignment_loss‚1.128377093392056 0loss/FBt1/assignment_loss‚0.2917424294302009 1loss/FBt2/assignment_loss‚0.18052271251144703 /loss/FRt2/detection_loss‚2.2112437314712086 1loss/SRqqt3/detection_loss‚1.6563385634921317 2loss/SRqqt1/assignment_loss‚2.4228220858744214 1loss/SRqqt2/detection_loss‚1.5648354255864698 1loss/FBt3/assignment_loss‚0.22849266509871086 /loss/FRt1/assignment_loss‚4.886929758389791 /loss/FRt4/assignment_loss‚3.763498959938685 _runtime‚6583.202633579 .loss/FBt4/detection_loss‚2.466875970363617 # _timestamp‚1.78841397279059e+09 2loss/SRqqt2/assignment_loss‚1.4631546349865534 /loss/FRt3/detection_loss‚1.8726291605886423 0loss/SRqqt4/detection_loss‚1.633267202342932Oò\ß!Ö 2loss/SRqqt2/assignment_loss‚1.4631546349865534 /loss/FRt3/detection_loss‚1.8726291605886423 0loss/SRqqt4/detection_loss‚1.633267202342932 1loss/SRqqt1/detection_loss‚2.5567270506705557 _step‚724 0loss/FRt2/assignment_loss‚3.2100103933077593 0loss/FBt4/assignment_loss‚0.6360510086566137 /loss/FRt4/detection_loss‚2.0989039917786916 /loss/FBt1/detection_loss‚2.4282903000712395 trainer/global_step‚8799 0loss/FRt3/assignment_loss‚2.5258544042425335 1loss/SRqqt4/assignment_loss‚1.499025277936688 .loss/FRt1/detection_loss‚2.919695858160655 .loss/FBt2/detection_loss‚3.462868893146515 .loss/FBt3/detection_loss‚0.751189925198206 epoch‚7 &loss/total_loss‚47.859347496636325 1loss/SRqqt3/assignment_loss‚1.128377093392056 0loss/FBt1/assignment_loss‚0.2917424294302009 1loss/FBt2/assignment_loss‚0.18052271251144703 /loss/FRt2/detection_loss‚2.2112437314712086 1loss/SRqqt3/detection_loss‚1.6563385634921317 2loss/SRqqt1/assignment_loss‚2.4228220858744214 1loss/SRqqt2/detection_loss‚1.5648354255864698 1loss/FBt3/assignment_loss‚0.22849266509871086 /loss/FRt1/assignment_loss‚4.886929758389791 /loss/FRt4/assignment_loss‚3.763498959938685 _runtime‚6583.202633579 .loss/FBt4/detection_loss‚2.466875970363617 # _timestamp‚1.78841397279059e+09ÔÞKŸ‘à!‚(:† –äÔÛ’Å™ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.557' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1114 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óÄ,/á!‚(:¤ –äÔ˜ð‰œ# proc.memory.rssMB‚ 2879.48828125" disk./.usageGB‚1097.8603515625 disk.drbd1777.out‚0 disk.rbd2.in‚0) proc.memory.percent‚17.57500171661377 cpu‚25.38490953637243) disk./.usagePercent‚73.67832935955025# disk.dm-0.out‚70572.39306640625 disk.rbd0.in‚0( proc.memory.availableMB‚ 249.23046875 disk.sda3.out‚ 260.46484375 disk.rbd3.in‚0 disk.rbd3.out‚0! disk.rbd2.out‚151.60302734375 disk.sda.in‚0 disk.dm-0.in‚ 711.06640625 disk.dm-1.in‚0 disk.dm-1.out‚ 260.46484375 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 260.46484375 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 3.435546875& disk.nvme0n1.out‚70572.39306640625 network.recv‚ 2.385436e+06 disk.nvme0n1.in‚ 711.06640625 proc.cpu.threads‚17 network.sent‚ 2.0942979e+07" disk.nvme0c0n1.in‚ 711.06640625( disk.nvme0c0n1.out‚70572.39306640625 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0# memory_percent‚98.4788179397583ZCpâ!‚(:… ¥äÔê™ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.424' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0848 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©i…«,ã!‚(:¡ ¥äÔÄôž disk.sda.out‚ 261.01171875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0& disk.nvme0c0n1.out‚70671.548828125% disk./.usageGB‚1097.8603324890137 disk.rbd2.in‚0) proc.memory.percent‚17.57500171661377 disk.rbd1.out‚ 3.435546875 disk.dm-0.in‚ 711.06640625$ memory_percent‚98.48036766052246" disk.nvme0c0n1.in‚ 711.06640625 disk.sda3.out‚ 261.01171875 disk.drbd1777.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.1051765e+07 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚ 711.06640625) disk./.usagePercent‚73.67832807951258 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0' proc.memory.availableMB‚ 248.9765625# proc.memory.rssMB‚ 2879.48828125 network.recv‚ 2.391873e+06 disk.dm-1.out‚ 261.01171875! disk.rbd2.out‚151.74365234375 disk.sda2.out‚0! disk.dm-0.out‚70671.548828125 disk.rbd3.out‚0$ disk.nvme0n1.out‚70671.548828125 cpu‚25.38417279085104­Â}‘ä!‚(:† ´äÔ°––™ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.508' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7016 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a’¾á-å!‚(:¢ ´äÔϘ¿› disk.nvme0n1.in‚ 711.06640625 disk.dm-1.out‚ 261.66015625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 261.66015625# proc.memory.rssMB‚ 2879.48828125# disk.dm-0.out‚70770.92919921875 disk.rbd3.in‚0 disk.sda2.in‚0) proc.memory.percent‚17.57500171661377$ disk./.usageGB‚1097.860336303711& disk.nvme0n1.out‚70770.92919921875 disk.sda.in‚0$ memory_percent‚98.47908020019531 cpu‚25.383400988602332 disk.rbd2.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚70770.92919921875 network.sent‚ 2.105713e+07 disk.rbd1.in‚0 disk.sda2.out‚0) disk./.usagePercent‚73.67832833552012 disk.dm-0.in‚ 711.06640625 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 711.06640625 disk.sda.out‚ 261.66015625 disk.sda1.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 3.435546875! disk.rbd2.out‚151.74365234375 disk.sda1.in‚0 disk.drbd1777.out‚0$ proc.memory.availableMB‚249.1875 network.recv‚ 2.39557e+06 disk.drbd1777.in‚0Ӭœ2æ!- _wandbruntime‚6617  _runtime‚6617Eß)@Çç!Á _runtime‚6617.174345285 _step‚725 & lr-AdamW/pg1‚2.054974766606474e-05 & lr-AdamW/pg2‚2.054974766606474e-05 trainer/global_step‚8849 % _timestamp‚1.7884140067621634e+09‰uæÌè!Æ & lr-AdamW/pg1‚2.054974766606474e-05 & lr-AdamW/pg2‚2.054974766606474e-05 trainer/global_step‚8849 % _timestamp‚1.7884140067621634e+09 _runtime‚6617.174345285 _step‚725Õ6H(2é!- _wandbruntime‚6617  _runtime‚6617^!–þí'ê!çO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚264535953297408 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116764909 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚6617.175234119 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12485346191872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178976718 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 % _timestamp‚1.7884140067629251e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178996756 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚277236912934400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50231072 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚262151434246656 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 178976718 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚262151247243776 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68161156 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚277237151653376 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12701174415872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚262151434246656 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12485177191936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12449191504214 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚249950982429394 trainer/global_step‚8849 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178996756 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12485177191936 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118392226 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12449356625618 _step‚726 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12485346191872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚249666088054784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116764909 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 295761665 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50231609 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚237501625803776 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12700959636992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68161154 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116764907 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116764907 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚249666070051840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚537 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚262151247243776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚249950800268630je™yŒë!ìO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1÷±µf'416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12485346191872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178976718 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 % _timestamp‚1.7884140067629251e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178996756 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚277236912934400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50231072 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚262151434246656 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 178976718 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚262151247243776 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68161156 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚277237151653376 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12701174415872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚262151434246656 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12485177191936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12449191504214 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚249950982429394 trainer/global_step‚8849 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178996756 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12485177191936 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118392226 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12449356625618 _step‚726 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12485346191872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚249666088054784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116764909 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 295761665 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50231609 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚237501625803776 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12700959636992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68161154 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116764907 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116764907 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚249666070051840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚537 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚262151247243776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚249950800268630 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚264535953297408 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116764909 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚6617.175234119Öª™ã÷2ì!- _wandbruntime‚6617  _runtime‚6617bk»÷&í!ñM bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12702586555392 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12450580668758 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178996505 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12702384372736 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 trainer/global_step‚8849 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚262180096762880 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12486739343360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚249693339416576 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 295774071 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚262180096762880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚249693339416576 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12486739343360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116777566 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12486570330112 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179016569 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226122752 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚249978314776482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12486570330112 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118404951 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6617.828087588 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179016569 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚264564866245632 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178996505 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68168533 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50236420 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚237527551946752 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 295794137 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116777566 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118405447 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚494 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 295794137 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚249693357419520 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚277267476741120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚264564890185728 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚237527568986112²…Ê^ü&î!öM E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116777566 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12486570330112 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179016569 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226122752 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚249978314776482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12486570330112 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118404951 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6617.828087588 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179016569 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚264564866245632 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178996505 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68168533 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50236420 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚237527551946752 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 295794137 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116777566 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118405447 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚494 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 295794137 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚249693357419520 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚277267476741120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚264564890185728 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚237527568986112 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12702586555392 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12450580668758 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178996505 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12702384372736 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 trainer/global_step‚8849 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚262180096762880 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12486739343360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚249693339416576 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 295774071 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚262180096762880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚249693339416576 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12486739343360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26×Þîz½2ï!- _wandbruntime‚6617  _runtime‚6617+Zð!Ô /loss/FBt2/detection_loss‚3.2423736651738486 /loss/FRt4/assignment_loss‚3.551741926566414 2loss/SRqqt3/assignment_loss‚1.5055093988776207 1loss/FBt3/assignment_loss‚0.19125261674758431 1loss/FBt4/assignment_loss‚0.18598635025294313 0loss/FRt3/assignment_loss‚3.3544612658225885 1loss/FBt2/assignment_loss‚0.05256535774181733 /loss/FBt3/detection_loss‚0.8777909760706519 epoch‚7 /loss/FRt4/detection_loss‚1.8896451398082654 1loss/SRqqt1/assignment_loss‚2.134173341772773 1loss/SRqqt3/detection_loss‚1.4511782919818705 /loss/FBt1/detection_loss‚3.1402730454098093 /loss/FBt4/detection_loss‚2.2588469844598036 0loss/FRt2/assignment_loss‚3.7092763923463368 /loss/FRt3/detection_loss‚1.9010898859824163 /loss/SRqqt1/detection_loss‚2.95001822439107 1loss/SRqqt2/detection_loss‚1.6541672166850832 /loss/FRt1/assignment_loss‚5.021110554536183 1loss/SRqqt2/assignment_loss‚1.238963654609742 .loss/FRt1/detection_loss‚2.985742670794328 0loss/SRqqt4/detection_loss‚1.639097731422495 _runtime‚6617.829402745 _step‚728 2loss/SRqqt4/assignment_loss‚1.7688540111261386 %loss/total_loss‚49.42236713432614 trainer/global_step‚8849 1loss/FBt1/assignment_loss‚0.25805617642535333 .loss/FRt2/detection_loss‚2.460192255321003 % _timestamp‚1.7884140074171808e+09ã¼ ¸ñ!Ù /loss/FBt3/detection_loss‚0.8777909760706519 epoch‚7 /loss/FRt4/detection_loss‚1.8896451398082654 1loss/SRqqt1/assignment_loss‚2.134173341772773 1loss/SRqqt3/detection_loss‚1.4511782919818705 /loss/FBt1/detection_loss‚3.1402730454098093 /loss/FBt4/detection_loss‚2.2588469844598036 0loss/FRt2/assignment_loss‚3.7092763923463368 /loss/FRt3/detection_loss‚1.9010898859824163 /loss/SRqqt1/detection_loss‚2.95001822439107 1loss/SRqqt2/detection_loss‚1.6541672166850832 /loss/FRt1/assignment_loss‚5.021110554536183 1loss/SRqqt2/assignment_loss‚1.238963654609742 .loss/FRt1/detection_loss‚2.985742670794328 0loss/SRqqt4/detection_loss‚1.639097731422495 _runtime‚6617.829402745 _step‚728 2loss/SRqqt4/assignment_loss‚1.7688540111261386 %loss/total_loss‚49.42236713432614 trainer/global_step‚8849 1loss/FBt1/assignment_loss‚0.25805617642535333 .loss/FRt2/detection_loss‚2.460192255321003 % _timestamp‚1.7884140074171808e+09 /loss/FBt2/detection_loss‚3.2423736651738486 /loss/FRt4/assignment_loss‚3.551741926566414 2loss/SRqqt3/assignment_loss‚1.5055093988776207 1loss/FBt3/assignment_loss‚0.1912526ÍgVœ§1674758431 1loss/FBt4/assignment_loss‚0.18598635025294313 0loss/FRt3/assignment_loss‚3.3544612658225885 1loss/FBt2/assignment_loss‚0.05256535774181733Ø,îËjò!‚(:„ ÃäÔìÀ¤™ gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.63' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äax.ó!‚(:£ ÃäÔ¥ïäœ disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd1777.in‚0 cpu‚25.38263923384525 disk.sda2.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0' disk.nvme0c0n1.out‚70870.3876953125 disk.dm-1.out‚ 262.37890625' proc.memory.availableMB‚ 248.9140625 disk.sda.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 3.435546875 disk.rbd2.in‚0) proc.memory.percent‚17.57500171661377 proc.cpu.threads‚17 network.recv‚ 2.399188e+06" disk.nvme0c0n1.in‚ 711.06640625$ memory_percent‚98.48074913024902 network.sent‚ 2.1062554e+07 disk.sda3.in‚0 disk.sda3.out‚ 262.37890625 disk.sda.out‚ 262.37890625 disk.dm-0.in‚ 711.06640625 disk.dm-1.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2879.48828125) disk./.usagePercent‚73.67832833552012 disk.nvme0n1.in‚ 711.06640625% disk.nvme0n1.out‚70870.3876953125" disk.dm-0.out‚70870.3876953125 disk.rbd0.in‚0! disk.rbd2.out‚153.38037109375$ disk./.usageGB‚1097.8603363037119›ˆ‘ô!‚(:† ÒäÔ’èÌ™ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.702' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5404 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J‹õ!‚(: ÒäÔƒˆ”œ# memory_percent‚98.4807014465332) proc.memory.percent‚17.57500171661377 disk.rbd3.out‚0 network.sent‚ 2.1171552e+07$ disk./.usageGB‚1097.860336303711 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 248.921875 disk.rbd1.in‚0 disk.dm-0.in‚ 711.06640625" disk.dm-0.out‚70871.0126953125! disk.rbd2.out‚153.39599609375 proc.cpu.threads‚17 disk.sda.in‚0 disk.dm-1.out‚262.75' disk.nvme0c0n1.out‚70871.0126953125" disk.nvme0c0n1.in‚ 711.06640625) disk./.usagePercent‚73.67832833552012 disk.nvme0n1.in‚ 711.06640625% disk.nvme0n1.out‚70871.0126953125 disk.dm-1.in‚0 cpu‚25.381896073239137 disk.sda.out‚262.75 disk.rbd0.in‚0 network.recv‚ 2.405904e+06 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2879.48828125 disk.sda3.out‚262.75 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 3.435546875 disk.rbd2.in‚0 disk.rbd3.in‚0•&2ö!- _wandbruntime‚6652  _runtime‚6652Us|Ç÷!Á ' lr-AdamW/pg1‚1.9965790959212182e-05 ' lr-AdamW/pg2‚1.9965790959212182e-05 trainer/global_step‚8899 $ _timestamp‚1.788414041619857e+09 _runtime‚ 6652.03198039 _step‚729sÀ#žÌø!Æ _runtime‚ 6652.03198039 _step‚729 ' lr-AdamW/pg1‚1.9965790959212182e-05 ' lr-AdamW/pg2‚1.9965790959212182e-05 trainer/global_step‚8899 $ _timestamp‚1.788414041619857e+09Ùd9‚2ù!- _wandbruntime‚6652  _runtime‚6652’&¡¨ì'ú!æO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117397859 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50500250 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚8899 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚278753445188096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12772059258368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚263584386803200 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117397857 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚567 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12518663322622 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12555017514496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚251029538288640 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 297387317 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119030256 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119029687 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251317431203838 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117397857 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788414041620585e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12554848514560 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179969420 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚278753683907072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚263584573806080 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _runtime‚6652.032938154 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179989458 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚238798767881216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚251029556291584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚569 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _step‚730 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 297367277 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚265981624648704 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12518828444026 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50499683 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚263584573806080 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12771844479488 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117397859 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50500250 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚8899 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚278753445188096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12772059258368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚263584386803200 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117397857 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚567 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12518663322622 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12555017514496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚251029538288640 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 297387317 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119030256 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119029687 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251317431203838 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117397857 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788414041620585e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12554848514560 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179969420 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚278753683907072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚263584573806080 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936Úø )B2ü!- _wandbruntime‚6652  _runtime‚6652Ó ¦ª!ý!ñM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚238824711063552 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117410516 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚251056807653376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179989207 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50505567 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 179989207 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 297419789 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 297399723 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12556410665984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚238824728102912 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12520217608778 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 _runtime‚6652.683935696 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚263613236322304 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68537383 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12520052487166 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12556241652736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚251344763550718 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117410518 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚266010513656832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12556410665984 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 % _timestamp‚1.7884140422712088e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12773267150848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚263613049306112 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119042431 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 180009271 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 trainer/global_step‚8899 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297419789 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚263613236322304 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12556241652736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0xf/^ü&þ!öM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12556410665984 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 % _timestamp‚1.7884140422712088e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12773267150848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚263613049306112 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119042431 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 180009271 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 trainer/global_step‚8899 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297419789 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚263613236322304 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12556241652736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚238824711063552 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117410516 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚251056807653376 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179989207 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50505567 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 179989207 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 297419789 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 297399723 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12556410665984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚238824728102912 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12520217608778 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 _runtime‚6652.683935696 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚263613236322304 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68537383 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12520052487166 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12556241652736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚251344763550718 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117410518 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚266010513656832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0یװ2ÿ!- _wandbruntime‚6652  _runtime‚6652­½aÄ[€"Õ /loss/FRt2/detection_loss‚2.0741631710635766 /loss/FBt1/detection_loss‚3.0300352796912193 0loss/FRt4/assignment_loss‚3.8352408307663937 1loss/SRqqt2/assignment_loss‚1.143972223876296 1loss/SRqqt3/detection_loss‚1.2660880078162466 /loss/FRt1/assignment_loss‚4.286671543121338 0loss/FBt2/assignment_loss‚0.3734002024160873 1loss/FBt3/assignment_loss‚0.16381136944823993 /loss/FRt3/detection_loss‚2.0841054835102777 % _timestamp‚1.7884140422735465e+09 /loss/FRt3/assignment_loss‚2.913846414197575 2loss/SRqqt4/assignment_loss‚1.8242350417375564 .loss/FRt1/detection_loss‚3.031278731425603 /loss/FBt2/detection_loss‚3.9554129123687742 _step‚732 2loss/SRqqt1/assignment_loss‚2.2133424137577866 1loss/SRqqt4/detection_loss‚1.7024819737672805 0loss/FBt4/assignment_loss‚0.5618012226769679 /loss/FBt3/detection_loss‚0.9264128138658716 trainer/global_step‚8899 0loss/FRt2/assignment_loss‚2.4292701727577617 2loss/SRqqt3/assignment_loss‚1.2221423842545067 .loss/FRt4/detection_loss‚2.218906783677162 1loss/SRqqt2/detection_loss‚1.4077759478744623 /loss/FBt4/detection_loss‚2.3244288459420206 %loss/total_loss‚47.80159311386979 _runtime‚6652.685636209 epoch‚7 0loss/FBt1/assignment_loss‚0.2916859762634279 1loss/SRqqt1/detection_loss‚2.5210833675933606Áàø`"Ú 2loss/SRqqt3/assignment_loss‚1.2221423842545067 .loss/FRt4/detection_loss‚2.218906783677162 1loss/SRqqt2/detection_loss‚1.4077759478744623 /loss/FBt4/detection_loss‚2.3244288459420206 %loss/total_loss‚47.80159311386979 _runtime‚6652.685636209 epoch‚7 0loss/FBt1/assignment_loss‚0.2916859762634279 1loss/SRqqt1/detection_loss‚2.5210833675933606 /loss/FRt2/detection_loss‚2.0741631710635766 /loss/FBt1/detection_loss‚3.0300352796912193 0loss/FRt4/assignment_loss‚3.8352408307663937 1loss/SRqqt2/assignment_loss‚1.143972223876296 1loss/SRqqt3/detection_loss‚1.2660880078162466 /loss/FRt1/assignment_loss‚4.286671543121338 0loss/FBt2/assignment_loss‚0.3734002024160873 1loss/FBt3/assignment_loss‚0.16381136944823993 /loss/FRt3/detection_loss‚2.0841054835102777 % _timestamp‚1.7884140422735465e+09 /loss/FRt3/assignment_loss‚2.913846414197575 2loss/SRqqt4/assignment_loss‚1.8242350417375564 .loss/FRt1/detection_loss‚3.031278731425603 /loss/FBt2/detection_loss‚3.9554129123687742 _step‚732 2loss/SRqqt1/assignment_loss‚2.2133424137577866 1loss/SRqqt4/detection_loss‚1.7024819737672805 0loss/FBt4/assignment_loss‚0.5618012226769679 /loss/FBt3/detection_loss‚0.9264128138658716 trainer/global_step‚8899 0loss/FRt2/assignment_loss‚2.4292701727577617Ü£1‚"‚(:… áäÔŠ‡¡š gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.235' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.647 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,<¼1ƒ"‚(:¦ áäÔÛ¼‰œ disk.sda.in‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚70970.18701171875( proc.memory.availableMB‚ 248.85546875) proc.memory.percent‚17.57500171661377 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 263.37109375 disk.drbd1777.in‚0 disk.rbd1.out‚ 3.435546875 disk.rbd2.out‚ 153.482421875" disk.nvme0c0n1.in‚ 711.06640625 cpu‚25.381114552493727 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-0.in‚ 711.06640625 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda2.out‚0) disk./.usagePercent‚73.67832833552012$ disk./.usageGB‚1097.860336303711# disk.dm-0.out‚70970.18701171875 disk.rbd0.in‚0( disk.nvme0c0n1.out‚70970.18701171875 network.sent‚ 2.1177731e+07 network.recv‚ 2.410407e+06 disk.drbd1777.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 263.37109375 disk.sda3.in‚0$ memory_percent‚98.48110675811768# proc.memory.rssMB‚ 2879.48828125 disk.sda.out‚ 263.37109375 disk.nvme0n1.in‚ 711.06640625 disk.sda2.in‚0ï#䜄"‚(:‘ ðäÔÙæ ™ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.594' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.918799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H`…(…"‚(: ðäÔÛø¡œ disk.rbd2.in‚0 disk.rbd2.out‚ 153.498046875 disk.sda1.out‚0$ memory_percent‚98.47967624664307# proc.memory.rssMB‚ 2879.48828125 network.sent‚ 2.1285791e+07) disk./.usagePercent‚73.67832833552012" disk.nvme0c0n1.in‚ 711.06640625 disk.sda2.in‚0$ disk./.usageGB‚1097.860336303711' disk.nvme0c0n1.out‚71069.8818359375 disk.sda2.out‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 249.08984375) proc.memory.percent‚17.57500171661377 disk.dm-1.out‚ 263.734375 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 711.06640625" disk.dm-0.out‚71069.8818359375 proc.cpu.threads‚17 disk.sda1.in‚0 cpu‚25.380409037821895 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.521484375% disk.nvme0n1.out‚71069.8818359375 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda.out‚ 263.734375 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 711.06640625 network.recv‚ 2.416248e+06 disk.dm-1.in‚0 disk.sda3.out‚ 263.734375JfÿI2†"- _wandbruntime‚6686  _runtime‚6686TÁþ|LJ"Á % _timestamp‚1.7884140763586328e+09 _runtime‚6686.770773601 _step‚733 & lr-AdamW/pg1‚1.938820115929869e-05 & lr-AdamW/pg2‚1.938820115929869e-05 trainer/global_step‚89493IržÌˆ"Æ & lr-AdamW/pg1‚1.938820115929869e-05 & lr-AdamW/pg2‚1.938820115929869e-05 trainer/global_step‚8949 % _timestamp‚1.7884140763586328e+09 _runtime‚6686.770773601 _step‚733ݯÊrò2‰"- _wandbruntime‚6686  _runtime‚6686§Oë™í'Š"çO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118030809 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252393006525440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12624519837184 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 180962122 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12624688837120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68898854 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 298992929 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119670022 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 180982160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚252393024528384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 180982160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚545 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚265017713365504 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118030807 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12842925239808 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 trainer/global_step‚8949 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240095944037376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚265017526362624 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚280270197299712 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚280269960677888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚267427248119808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚265017713365504 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚547 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68898856 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚267427272059904 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚252393024528384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 180962122 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 299012969 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50771713 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚6686.771768465 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118030809 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚252684244299810 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240095926998016 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12588300262434 % _timestamp‚1.7884140763594322e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚265017526362624 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚252393006525440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12624519837184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224ÍÚ^Ý‹"ìO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12624688837120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68898854 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 298992929 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119670022 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 180982160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚252393024528384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 180982160 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚545 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚265017713365504 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118030807 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12842925239808 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 trainer/global_step‚8949 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240095944037376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚265017526362624 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚280270197299712 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚280269960677888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚267427248119808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚265017713365504 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚547 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68898856 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚267427272059904 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚252393024528384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 180962122 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 299012969 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50771713 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚6686.771768465 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118030809 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚252684244299810 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240095926998016 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12588300262434 % _timestamp‚1.7884140763594322e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚265017526362624 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚252393006525440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12624519837184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118030809 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252393006525440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12624519837184 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 180962122 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚502 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚267456161068032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884140770222864e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118043466 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12625912975360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12589524305574 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299045441 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50776596 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12844133705728 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119682827 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚267456185008128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚252711394485926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚280300294773760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12589689427186 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚8949 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚252420275890176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚240121870180352 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚265046188865536 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚265046375881728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 299045441 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 299025375 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6687.434962825 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12626081988608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚240121887219712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚252420275890176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚502 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚267456161068032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884140770222864e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118043466 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12625912975360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12589524305574 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299045441 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50776596 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12844133705728 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119682827 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚267456185008128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚252711394485926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚280300294773760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12589689427186 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚8949 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚252420275890176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚240121870180352 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚265046188865536 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚265046375881728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 299045441 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 299025375 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6687.434962825 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12626081988608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚240121887219712ß—/ u2"- _wandbruntime‚6687  _runtime‚66874Mƒ\X"Ò 2loss/SRqqt4/assignment_loss‚1.6044011306509656 /loss/FRt1/detection_loss‚3.0453707432746886 0loss/SRqqt1/detection_loss‚2.497184258082817 /loss/FBt1/detection_loss‚3.1516804099082947 %loss/total_loss‚47.35682253299042 0loss/FRt3/assignment_loss‚2.7634792799173398 epoch‚7 _step‚736 0loss/FRt4/assignment_loss‚3.5998126916263415 2loss/SRqqt1/assignment_loss‚2.2876812825942863 0loss/FBt2/assignment_loss‚0.5733008011011407 /loss/FBt2/detection_loss‚2.3064251095056534 2loss/SRqqt3/assignment_loss‚1.7749640616277853 .loss/FBt4/detection_loss‚2.478251381715139 /loss/FRt1/assignment_loss‚4.158984265327454 .loss/FRt3/detection_loss‚2.215449570569881 /loss/FRt4/detection_loss‚1.8276189688755156 /loss/FBt3/detection_loss‚0.6442716456878491 trainer/global_step‚8949 % _timestamp‚1.7884140770241659e+09 2loss/SRqqt2/assignment_loss‚1.6182042182473517 0loss/FBt1/assignment_loss‚0.4226754271403479 0loss/FBt3/assignment_loss‚0.3386804791472594 0loss/SRqqt3/detection_loss‚1.547925154864788 _runtime‚6687.436264917 1loss/FBt4/assignment_loss‚0.43230907918295997 0loss/FRt2/assignment_loss‚3.0937757117132985 .loss/FRt2/detection_loss‚1.970266661336345 0loss/SRqqt2/detection_loss‚1.304376883432269 0loss/SRqqt4/detection_loss‚1.6997333174606542JnT]‘"× /loss/FBt1/detection_loss‚3.1516804099082947 %loss/total_loss‚47.35682253299042 0loss/FRt3/assignment_loss‚2.7634792799173398 epoch‚7 _step‚736 0loss/FRt4/assignment_loss‚3.5998126916263415 2loss/SRqqt1/assignment_loss‚2.2876812825942863 0loss/FBt2/assignment_loss‚0.5733008011011407 /loss/FBt2/detection_loss‚2.3064251095056534 2loss/SRqqt3/assignment_loss‚1.7749640616277853 .loss/FBt4/detection_loss‚2.478251381715139 /loss/FRt1/assignment_loss‚4.158984265327454 .loss/FRt3/detection_loss‚2.215449570569881 /loss/FRt4/detection_loss‚1.8276189688755156 /loss/FBt3/detection_loss‚0.6442716456878491 trainer/global_step‚8949 % _timestamp‚1.7884140770241659e+09 2loss/SRqqt2/assignment_loss‚1.6182042182473517 0loss/FBt1/assignment_loss‚0.4226754271403479 0loss/FBt3/assignment_loss‚0.3386804791472594 0loss/SRqqt3/detection_loss‚1.547925154864788 _runtime‚6687.436264917 1loss/FBt4/assignment_loss‚0.43230907918295997 0loss/FRt2/assignment_loss‚3.0937757117132985 .loss/FRt2/detection_loss‚1.970266661336345 0loss/SRqqt2/detection_loss‚1.304376883432269 0loss/SRqqt4/detection_loss‚1.699733317460654 2loss/SRqqt4/assignment_loss‚1.6044011306509656 /loss/FRt1/detection_loss‚3.0453707432746886 0loss/SRqqt1/detection_loss‚2.497184258082817à]¨™J’"‚(:… ÿäÔ•Ô¿› gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.892' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3784 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨ÖÔð+“"‚(:  ÿäÔ¥ÕÖ' disk.nvme0c0n1.out‚71168.9599609375( proc.memory.availableMB‚ 248.83984375 disk.drbd1777.in‚0 disk.rbd1.out‚ 3.583984375) disk./.usagePercent‚73.67834036787423% disk./.usageGB‚1097.8605155944824 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 711.06640625" disk.dm-0.out‚71168.9599609375 disk.dm-1.out‚ 264.5390625" disk.nvme0c0n1.in‚ 711.06640625# proc.memory.rssMB‚ 2879.48828125 disk.sda3.out‚ 264.5390625 disk.sda.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0$ memory_percent‚98.48120212554932 cpu‚25.37965985394031 disk.rbd2.in‚0 disk.drbd1777.out‚0 network.recv‚ 2.420103e+06 disk.sda.out‚ 264.5390625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0) proc.memory.percent‚17.57500171661377 network.sent‚ 2.1291291e+07% disk.nvme0n1.out‚71168.9599609375 disk.dm-0.in‚ 711.06640625 disk.rbd2.out‚ 153.623046875 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd3.out‚0Êðû,›”"‚(: Ž‘äÔÚ㟙 gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.563' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.712600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$2Û1•"‚(:¦ Ž‘äÔþÀÌœ disk.sda.out‚ 265.05078125) disk./.usagePercent‚73.67833908783655 disk.dm-0.in‚ 711.06640625# disk.dm-0.out‚71169.31787109375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0 network.sent‚ 2.1400158e+07 disk.rbd3.in‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 711.06640625 network.recv‚ 2.426609e+06 disk.rbd3.out‚0 disk.sda3.out‚ 265.05078125 disk.rbd1.out‚ 3.583984375) proc.memory.percent‚17.57500171661377 disk.rbd2.in‚0$ memory_percent‚98.48053455352783# proc.memory.rssMB‚ 2879.48828125 proc.cpu.threads‚17 disk.dm-1.out‚ 265.05078125 disk.drbd1777.out‚0& disk.nvme0n1.out‚71169.31787109375( proc.memory.availableMB‚ 248.94921875 cpu‚25.378933507782442 disk.rbd2.out‚ 155.150390625$ disk./.usageGB‚1097.860496520996 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚71169.31787109375 disk.rbd1.in‚0 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 711.06640625 disk.sda.in‚0£¹Ñ–"‚(:„ ‘äÔó—³™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.63' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Še?--—"‚(:¢ ‘äÔËñ¢œ disk.sda2.out‚0 disk.drbd1777.out‚0) proc.memory.percent‚17.57500171661377 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 cpu‚25.37820449268106$ disk./.usageGB‚1097.860496520996 disk.nvme0n1.in‚ 711.06640625 disk.dm-1.out‚ 265.57421875& disk.nvme0n1.out‚71268.41943359375# disk.dm-0.out‚71268.41943359375 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 711.06640625 disk.sda3.in‚0# memory_percent‚98.4804630279541 disk.rbd0.in‚0 disk.rbd2.out‚ 155.212890625 disk.sda3.out‚ 265.57421875 disk.sda.out‚ 265.57421875 disk.sda1.in‚0 network.sent‚ 2.14064e+07 disk.rbd1.out‚ 3.6162109375' proc.memory.availableMB‚ 248.9609375 disk.dm-0.in‚ 711.06640625 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0# proc.memory.rssMB‚ 2879.48828125 network.recv‚ 2.431418e+06) disk./.usagePercent‚73.67833908783655 disk.rbd1.in‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚71268.41943359375‡ä†·2˜"- _wandbruntime‚6721  _runtime‚6721Oø4}éJÐÈ™"à % _timestamp‚1.7884141112989256e+09 _runtime‚6721.711232257 _step‚737 ' lr-AdamW/pg1‚1.8817098635172885e-05 ' lr-AdamW/pg2‚1.8817098635172885e-05 trainer/global_step‚89994ûBJΚ"È % _timestamp‚1.7884141112989256e+09 _runtime‚6721.711232257 _step‚737 ' lr-AdamW/pg1‚1.8817098635172885e-05 ' lr-AdamW/pg2‚1.8817098635172885e-05 trainer/global_step‚8999áó“ý2›"- _wandbruntime‚6721  _runtime‚67211Jeaí'œ"çO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118663759 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12694360159744 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12913688477696 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181974862 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181954824 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚254050875235018 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚266450665922048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚281786820630528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚253756492765184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚281786584008704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚573 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118663759 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚253756474762240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69267704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181954824 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120310747 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚253756492765184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 300638621 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12657772080842 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120310174 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚268872919471104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12694360159744 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118663757 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 % _timestamp‚1.7884141112998357e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 300618581 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51043041 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 trainer/global_step‚8999 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚266450852924928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69267706f­Gxò'"ìO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118663757 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 % _timestamp‚1.7884141112998357e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 300618581 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51043041 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 trainer/global_step‚8999 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚266450852924928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69267706 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118663759 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12694360159744 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12913688477696 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181974862 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181954824 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚254050875235018 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚266450665922048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚281786820630528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚253756492765184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚281786584008704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚573 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118663759 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚253756474762240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69267704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181954824 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120310747 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚253756492765184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 300638621 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12657772080842 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120310174 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚268872919471104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12694360159744â2ž"- _wandbruntime‚6722  _runtime‚6722.QH÷&Ÿ"ñM _runtime‚6722.359221892 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181994675 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚266479328424960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚268901832419328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12915320491520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚254078207582106 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300651027 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚254078025421134 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12695584297984 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12695584297984 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚515 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300671093 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300671093 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12658996123982 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118676416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51047923 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120323004 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12695753311232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚253783744126976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚281816920496640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚241419029297152 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚253783744126976 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118676418 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚281817152910848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300651027 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12695753311232 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12915112017408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12659161245594 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7884141119465697e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118676416 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 trainer/global_step‚8999 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192°¹ø "öM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚254078025421134 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12695584297984 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12695584297984 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚515 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300671093 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_byte?©U s.all.current‚ 182160972 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300671093 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12658996123982 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118676416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51047923 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120323004 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12695753311232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚253783744126976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚281816920496640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚241419029297152 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚253783744126976 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118676418 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚281817152910848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300651027 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12695753311232 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12915112017408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12659161245594 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7884141119465697e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118676416 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 trainer/global_step‚8999 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 _runtime‚6722.359221892 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181994675 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚266479328424960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚268901832419328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12915320491520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚254078207582106 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300651027 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064ã<Ø(Ý2¡"- _wandbruntime‚6722  _runtime‚6722Ìe•aZ¢"Ô % _timestamp‚1.7884141119482872e+09 1loss/SRqqt1/assignment_loss‚2.764256364107132 .loss/FRt4/detection_loss‚2.070259388755376 1loss/SRqqt4/detection_loss‚1.6773432723500512 _runtime‚6722.360630946 /loss/FRt1/assignment_loss‚4.245293305470393 2loss/SRqqt3/assignment_loss‚1.7139157551074666 1loss/SRqqt4/assignment_loss‚1.573240458287976 /loss/FBt1/detection_loss‚2.6808966398239136 0loss/FRt3/assignment_loss‚2.4979347858512613 1loss/SRqqt2/detection_loss‚1.4328728247256506 epoch‚7 /loss/FBt3/detection_loss‚0.7789724847528162 .loss/FBt4/detection_loss‚2.494706396873181 trainer/global_step‚8999 0loss/FRt4/assignment_loss‚3.3275425071809805 1loss/SRqqt2/assignment_loss‚1.511057288622454 0loss/FBt1/assignment_loss‚0.2958310119473026 1loss/FBt2/assignment_loss‚0.15664528973219502 0loss/FBt3/assignment_loss‚0.2419976559530188 /loss/FRt3/detection_loss‚1.9102743375196791 /loss/FRt2/detection_loss‚2.4020603992218197 1loss/SRqqt1/detection_loss‚3.0993931621313093 /loss/FBt2/detection_loss‚2.6586206555366516 /loss/FRt1/detection_loss‚2.9833874060557437 1loss/SRqqt3/detection_loss‚1.3360923423564859 _step‚740 /loss/FRt2/assignment_loss‚2.756978434185649 1loss/FBt4/assignment_loss‚0.31185749769037313 %loss/total_loss‚46.92142966423888ÍŠ+´_£"Ù 1loss/SRqqt2/detection_loss‚1.4328728247256506 epoch‚7 /loss/FBt3/detection_loss‚0.7789724847528162 .loss/FBt4/detection_loss‚2.494706396873181 trainer/global_step‚8999 0loss/FRt4/assignment_loss‚3.3275425071809805 1loss/SRqqt2/assignment_loss‚1.511057288622454 0loss/FBt1/assignment_loss‚0.2958310119473026 1loss/FBt2/assignment_loss‚0.15664528973219502 0loss/FBt3/assignment_loss‚0.2419976559530188 /loss/FRt3/detection_loss‚1.9102743375196791 /loss/FRt2/detection_loss‚2.4020603992218197 1loss/SRqqt1/detection_loss‚3.0993931621313093 /loss/FBt2/detection_loss‚2.6586206555366516 /loss/FRt1/detection_loss‚2.9833874060557437 1loss/SRqqt3/detection_loss‚1.3360923423564859 _step‚740 /loss/FRt2/assignment_loss‚2.756978434185649 1loss/FBt4/assignment_loss‚0.31185749769037313 %loss/total_loss‚46.92142966423888 % _timestamp‚1.7884141119482872e+09 1loss/SRqqt1/assignment_loss‚2.764256364107132 .loss/FRt4/detection_loss‚2.070259388755376 1loss/SRqqt4/detection_loss‚1.6773432723500512 _runtime‚6722.360630946 /loss/FRt1/assignment_loss‚4.245293305470393 2loss/SRqqt3/assignment_loss‚1.7139157551074666 1loss/SRqqt4/assignment_loss‚1.573240458287976 /loss/FBt1/detection_loss‚2.6808966398239136 0loss/FRt3/assignment_loss‚2.4979347858512613äú¬px›¤"‚(: ¬‘äÔŸÒ–™ gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚91.895' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.378999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!§ø*¥"‚(:Ÿ ¬‘äÔ–”žœ disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.percent‚17.57500171661377 cpu‚25.377439483977607$ disk./.usageGB‚1097.860553741455 disk.rbd1.out‚ 3.6162109375 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2879.48828125" disk.nvme0c0n1.in‚ 711.06640625$ memory_percent‚98.48177433013916 network.recv‚ 2.4354e+06 disk.dm-0.in‚ 711.06640625 disk.rbd3.out‚0' disk.nvme0c0n1.out‚71367.6591796875" disk.dm-0.out‚71367.6591796875 disk.sda3.out‚ 266.0546875) disk./.usagePercent‚73.67834292794957 disk.nvme0n1.in‚ 711.06640625% disk.nvme0n1.out‚71367.6591796875 disk.dm-1.in‚0 disk.dm-1.out‚ 266.0546875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 network.sent‚ 2.1412076e+07 disk.rbd2.out‚ 155.228515625 disk.sda2.in‚0 disk.sda.out‚ 266.0546875( proc.memory.availableMB‚ 248.74609375 proc.cpu.threads‚17.ã0¦"‚(:… »‘äÔ³Úš™ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚91.932' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.3864 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·é„§"‚(:‘ »‘äÔÕ êœ disk.nvme0n1.in‚ 711.06640625" disk.nvme0c0n1.in‚ 711.06640625 cpu‚25.37667308927042 proc.cpu.threads‚17 disk.rbd2.out‚ 155.291015625 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 network.sent‚ 2.1520486e+07% disk./.usageGB‚1097.8605499267578 disk.dm-0.out‚ 71466.8046875 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0 disk.sda.out‚ 266.68359375 proc.memory.availableMB‚249 disk.rbd2.in‚0 disk.sda3.out‚ 266.68359375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.out‚0) proc.memory.percent‚17.57500171661377) disk./.usagePercent‚73.67834267194203" disk.nvme0n1.out‚ 71466.8046875 disk.sda2.in‚0 disk.rbd1.out‚ 3.67919921875# proc.memory.rssMB‚ 2879.48828125 network.recv‚ 2.441143e+06 disk.dm-0.in‚ 711.06640625 disk.dm-1.out‚ 266.68359375 disk.rbd0.out‚0$ disk.nvme0c0n1.out‚ 71466.8046875 disk.sda.in‚0 disk.drbd1777.in‚0" memory_percent‚98.480224609375œ2¨"- _wandbruntime‚6756  _runtime‚6756í9ZŸÈ©" $ _timestamp‚1.788414145826363e+09 _runtime‚6756.238499804 _step‚741 ' lr-AdamW/pg1‚1.8252602403744777e-05 ' lr-AdamW/pg2‚1.8252602403744777e-05 trainer/global_step‚9049µäy5ͪ"Ç ' lr-AdamW/pg1‚1.8252602403744777e-05 ' lr-AdamW/pg2‚1.8252602403744777e-05 trainer/global_step‚9049 $ _timestamp‚1.788414145826363e+09 _runtime‚6756.238499804 _step‚741åšÒ(Ö2«"- _wandbruntime‚6756  _runtime‚6756aŽLÛí'¬"çO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 302244233 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚255119961001984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12763862482432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚537 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119296709 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 trainer/global_step‚9049 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69636556 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267883805481472 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚242690262270976 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚255119961001984 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120950118 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69636554 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12764031482368 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚270318566882304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 % _timestamp‚1.7884141458271651e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚283303382905856 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 182947526 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51313564 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12984605438976 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12727243899250 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 182947526 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267883992484352 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚283303148381184 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182967564 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 302264273 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚242690245231616 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267883992484352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119296707 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚255119942999040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚270318542942208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚6756.239510347 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119296709 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚267883805481472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12984816023552 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12763862482432 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚255417324009462 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51313564 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12984605438976 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12727243899250 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 182947526 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267883992484352 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚283303148381184 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182967564 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 302264273 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚242690245231616 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267883992484352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119296707 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚255119942999040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚270318542942208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚6756.239510347 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119296709 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚267883805481472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12984816023552 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12763862482432 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚255417324009462 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267883805481472 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚242690262270976 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚255119961001984 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120950118 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69636554 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12764031482368 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚270318566882304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 % _timestamp‚1.7884141458271651e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚283303382905856 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408æKƒ 2®"- _wandbruntime‚6756  _runtime‚6756ʸDåö&¯"ðM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12728633064002 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12765255620608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚267912655000576 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚255147230366720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267912655000576 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 182967313 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119309368 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 182987377 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119309368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12986234941952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302296745 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51318907 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚267912467984384 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 182967313 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69643931 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12986028564992 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12728467942390 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69643933 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚255147230366720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119309366 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚242716188413952 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120962838 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 % _timestamp‚1.7884141464867768e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119309366 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚242716188413952 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120962838 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 % _timestamp‚1.7884141464867768e+09 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚255147230366720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267912655000576 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 182967313 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119309368 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 182987377 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119309368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12986234941952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302296745 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51318907 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚267912467984384 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 182967313 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69643931 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12986028564992çž;±*2±"- _wandbruntime‚6756  _runtime‚6756mwëX²"Ò 1loss/FBt3/assignment_loss‚0.23569815692477836 .loss/FRt2/detection_loss‚2.181684337687074 /loss/FRt3/detection_loss‚1.7156539057882934 0loss/SRqqt1/detection_loss‚2.592789809703827 0loss/SRqqt4/detection_loss‚1.729839939820139 _step‚744 trainer/global_step‚9049 1loss/SRqqt2/assignment_loss‚1.506239839973794 /loss/FRt4/detection_loss‚2.1021447252143513 /loss/FBt1/detection_loss‚3.1451499313116074 /loss/FBt3/detection_loss‚0.8463548672373725 &loss/total_loss‚47.199902440209016 epoch‚7 2loss/SRqqt4/assignment_loss‚1.4295347861030645 1loss/SRqqt3/detection_loss‚1.4177250050685621 /loss/FRt2/assignment_loss‚2.990499225624821 2loss/SRqqt3/assignment_loss‚1.4241651739586483 % _timestamp‚1.7884141464883716e+09 /loss/FRt1/detection_loss‚3.1546589303016663 1loss/SRqqt2/detection_loss‚1.5487129757521858 /loss/FBt2/detection_loss‚2.9952018782496452 0loss/FRt3/assignment_loss‚2.0353495616179247 0loss/FBt1/assignment_loss‚0.5028800724406514 0loss/FBt2/assignment_loss‚0.2766388269592426 /loss/FBt4/assignment_loss‚0.578727143746424 /loss/FRt1/assignment_loss‚4.859498062133789 0loss/FRt4/assignment_loss‚3.5656677051024004 0loss/SRqqt1/assignment_loss‚1.99814603805542 /loss/FBt4/detection_loss‚2.3669415414333344 _runtime‚6756.900387636‰%OH]³"× /loss/FBt4/detection_loss‚2.3669415414333344 _runtime‚6756.900387636 1loss/FBt3/assignment_loss‚0.23569815692477836 .loss/FRt2/detection_loss‚2.181684337687074 /loss/FRt3/detection_loss‚1.7156539057882934 0loss/SRqqt1/detection_loss‚2.592789809703827 0loss/SRqqt4/detection_loss‚1.729839939820139 _step‚744 trainer/global_step‚9049 1loss/SRqqt2/assignment_loss‚1.506239839973794 /loss/FRt4/detection_loss‚2.1021447252143513 /loss/FBt1/detection_loss‚3.1451499313116074 /loss/FBt3/detection_loss‚0.8463548672373725 &loss/total_loss‚47.199902440209016 epoch‚7 2loss/SRqqt4/assignment_loss‚1.4295347861030645 1loss/SRqqt3/detection_loss‚1.4177250050685621 /loss/FRt2/assignment_loss‚2.990499225624821 2loss/SRqqt3/assignment_loss‚1.4241651739586483 % _timestamp‚1.7884141464883716e+09 /loss/FRt1/detection_loss‚3.1546589303016663 1loss/SRqqt2/detection_loss‚1.5487129757521858 /loss/FBt2/detection_loss‚2.9952018782496452 0loss/FRt3/assignment_loss‚2.0353495616179247 0loss/FBt1/assignment_loss‚0.5028800724406514 0loss/FBt2/assignment_loss‚0.2766388269592426 /loss/FBt4/assignment_loss‚0.578727143746424 /loss/FRt1/assignment_loss‚4.859498062133789 0loss/FRt4/assignment_loss‚3.5656677051024004 0loss/SRqqt1/assignment_loss‚1.99814603805542è°‘´"‚(:† Ê‘äÔ¨¥·™ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.304' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8608 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚â3µ"‚(:¨ Ê‘äÔáø’œ disk.drbd1777.in‚0& disk.nvme0n1.out‚71467.34619140625" disk.nvme0c0n1.in‚ 711.06640625 network.sent‚ 2.1526741e+07 network.recv‚ 2.445807e+06 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚ 711.06640625# proc.memory.rssMB‚ 2879.48828125 disk.dm-1.out‚ 267.04296875 disk.drbd1777.out‚0# disk.dm-0.out‚71467.34619140625) proc.memory.percent‚17.57500171661377 cpu‚25.37595656516215 disk.rbd1.in‚0 disk.rbd1.out‚ 3.67919921875 disk.sda1.in‚0 disk.rbd3.out‚0) disk./.usagePercent‚73.67834446399478% disk./.usageGB‚1097.8605766296387 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.sda.out‚ 267.04296875 proc.cpu.threads‚17 disk.rbd0.in‚0! disk.rbd2.out‚155.39306640625 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-0.in‚ 711.06640625 disk.rbd3.in‚0( disk.nvme0c0n1.out‚71467.34619140625 disk.sda3.out‚ 267.04296875& proc.memory.availableMB‚ 249.203125$ memory_percent‚98.47898483276367 ùú‘¶"‚(:† Ù‘äÔ¦“š gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚122.034' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷~…ª*·"‚(:Ÿ Ù‘äÔ’ú˜ disk.drbd1777.out‚0 disk.rbd1.in‚0! disk.rbd2.out‚155.39306640625 disk.rbd3.out‚0 disk.sda3.out‚ 267.7578125) proc.memory.percent‚17.57500171661377% disk./.usageGB‚1097.8608474731445& disk.nvme0n1.out‚71566.44384765625# disk.dm-0.out‚71566.44384765625 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda3.in‚0$ proc.memory.availableMB‚248.8125 disk.dm-1.out‚ 267.7578125 disk.sda1.out‚0# proc.memory.rssMB‚ 2879.48828125 network.recv‚ 2.452276e+06 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 267.7578125 disk.sda1.in‚0 network.sent‚ 2.1635576e+07 disk.nvme0n1.in‚ 711.06640625 disk.sda2.in‚0( disk./.usagePercent‚73.6783626405297 disk.rbd1.out‚ 3.7421875 disk.dm-0.in‚ 711.06640625" disk.nvme0c0n1.in‚ 711.06640625 disk.sda.in‚0 cpu‚25.375266565339956( disk.nvme0c0n1.out‚71566.44384765625$ memory_percent‚98.48136901855469 disk.drbd1777.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17bà2¸"- _wandbruntime‚6791  _runtime‚6791Ýƹ"À $ _timestamp‚1.788414180597685e+09 _runtime‚6791.009858132 _step‚745 & lr-AdamW/pg1‚1.769483010518292e-05 & lr-AdamW/pg2‚1.769483010518292e-05 trainer/global_step‚9099{˜t˺"Å & lr-AdamW/pg1‚1.769483010518292e-05 & lr-AdamW/pg2‚1.769483010518292e-05 trainer/global_step‚9099 $ _timestamp‚1.788414180597685e+09 _runtime‚6791.009858132 _step‚745é ª2»"- _wandbruntime‚6791  _runtime‚6791½´Íšx¼"çO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚269317132043776 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13055510404096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12796550596254 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12833533805056 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚269316945040896 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 303889925 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70005404 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183960266 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚256783954944670 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119929657 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51585450 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚269316945040896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12833702804992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70005406 _step‚746 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚509 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119929657 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 303869885 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚284819928990720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121590345 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12833702804992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚511 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚284819700757504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12833533805056 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚269317132043776 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚256483429238784 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119929659 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 183940228 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚271764214293504 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚256483411235840 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚256483429238784 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚243987404348416 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚243987421387776 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 _runtime‚6791.010798003CB?ò'½"ìO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12833533805056 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚269316945040896 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 303889925 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70005404 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183960266 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚256783954944670 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119929657 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51585450 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚269316945040896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12833702804992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70005406 _step‚746 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚509 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119929657 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 303869885 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚284819928990720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121590345 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12833702804992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚511 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚284819700757504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12833533805056 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚269317132043776 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚256483429238784 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119929659 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 183940228 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚271764214293504 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚256483411235840 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚256483429238784 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚243987404348416 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚243987421387776 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 _runtime‚6791.010798003 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚269317132043776 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13055510404096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12796550596254êŠæ¢u2¾"- _wandbruntime‚6791  _runtime‚6791îœRÿ÷&¿"ñM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚256811469452522 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚256510698603520 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12797939760798 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚269345607543808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 303902331 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183980079 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚530 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚271793127241728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚269345794560000 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121602993 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13057140302336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 303902331 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚271793103301632 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119942316 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13056921342464 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183980079 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121603525 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6791.665894177 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚244013364570112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70012781 _step‚747 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚284850024644096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚244013347530752 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 303922397 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119942318 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚256510680600576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51590212 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚256510680600576 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚256811287291550 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 303922397 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51590742 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 % _timestamp‚1.7884141812532206e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12835095956480 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12834926943232 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183960015 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 trainer/global_step‚9099 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119942316 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12835095956480æ´ý MÀ"öM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚284850267544064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183980079 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚530 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚271793127241728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚269345794560000 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121602993 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13057140302336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 303902331 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚271793103301632 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119942316 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13056921342464 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183980079 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121603525 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6791.665894177 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚244013364570112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70012781 _step‚747 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚284850024644096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚244013347530752 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 303922397 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119942318 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚256510680600576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51590212 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚256510680600576 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚256811287291550 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 303922397 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51590742 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 % _timestamp‚1.7884141812532206e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12835095956480 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12834926943232 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183960015 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 trainer/global_step‚9099 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119942316 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12835095956480 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚256811469452522 KEDeviceStatsMonitor.on_train_õº|Я batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚256510698603520 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12797939760798 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚269345607543808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 303902331 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚257846897475584 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51853766 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚286336507247616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120562607 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70374254 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚270750084600320 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12903374127616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚518 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚257846897475584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 122228540 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚750 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚257846879472640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12866022414662 _runtime‚6825.613476779 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120562609¹ôháò'Ï"ìO RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 122228540 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚750 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚257846879472640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12866022414662 _runtime‚6825.613476779 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120562609 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184932930 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚257846879472640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚286336272722944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12903205127680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚258150768040642 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚257846897475584 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51853766 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚286336507247616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120562607 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70374254 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚270750084600320 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12903374127616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚518 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚257846897475584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248îãžë±2Ð"- _wandbruntime‚6826  _runtime‚6826ùðúÿÊÑ"ðM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚257874148837376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122241227 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚270778934119424 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚286366836988416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12867411579206 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120575268 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 trainer/global_step‚9149 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 MGDÐ=I,#eviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚270778934119424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚273238750712832 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12904598265856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70381633 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184972781 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚257874166840320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51859594 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13127849667072 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 305527983 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚257874166840320 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚245310523686912 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12867576700818 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120575266 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 305548049 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚519 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚270778747103232 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚286366600379904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12904767279104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120575266 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70381631 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 305527983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6826.270285472 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚245310506647552 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 184952717 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12904598265856 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 184972781 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 $ _timestamp‚1.788414215857365e+09 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305548049 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚273238774652928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122240706 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚258177918226758 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚521 _step‚751F®ÿÃû&Ò"õM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚521 _step‚751 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚257874148837376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122241227 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚270778934119424 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚286366836988416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12867411579206 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120575268 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 trainer/global_step‚9149 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚270778934119424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚273238750712832 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12904598265856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70381633 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184972781 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚257874166840320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51859594 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13127849667072 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 305527983 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚257874166840320 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚245310523686912 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12867576700818 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120575266 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 305548049 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚519 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚270778747103232 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚286366600379904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12904767279104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120575266 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70381631 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 305527983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6826.270285472 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚245310506647552 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 184952717 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12904598265856 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 184972781 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 $ _timestamp‚1.788414215857365e+09 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305548049 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚273238774652928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122240706 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚258177918226758 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416ï—érû2Ó"- _wandbruntime‚6826  _runtime‚6826ÿ±,XÔ"Ò 2loss/SRqqt4/assignment_loss‚1.5773876555786843 0loss/FBt1/assignment_loss‚0.5850794333604427 /loss/FBt4/assignment_loss‚0.199735587764917 /loss/FBt1/detection_loss‚2.9883818924427032 /loss/FBt4/detection_loss‚3.2217600643634796 0loss/FBt2/assignment_loss‚0.4846670716917288 /loss/FRt1/assignment_loss‚4.273233210339265 /loss/FRt2/assignment_loss‚2.196493243177732 /loss/FRt2/detection_loss‚2.1190173445890346 0loss/FRt3/assignment_loss‚2.0850328219180203 .loss/FRt1/detection_loss‚2.782246933263891 /loss/FBt2/detection_loss‚3.3307902216911316 $ _timestamp‚1.788414215859538e+09 _runtime‚6826.271603385 0loss/FRt4/assignment_loss‚3.3292494817421985 1loss/SRqqt1/assignment_loss‚2.511211797595024 /loss/FRt4/detection_loss‚2.2503106582623262 /loss/FBt3/detection_loss‚0.7487716392834526 trainer/global_step‚9149 1loss/FBt3/assignment_loss‚0.19461381063221575 /loss/FRt3/detection_loss‚1.9572393267738575 /loss/SRqqt1/detection_loss‚2.46351569550378 1loss/SRqqt3/detection_loss‚1.2069374557584525 &loss/total_loss‚47.110281134062745 _step‚752 2loss/SRqqt2/assignment_loss‚1.5266957243935007 1loss/SRqqt3/assignment_loss‚1.652943461885055 1loss/SRqqt2/detection_loss‚1.6031253919249675 1loss/SRqqt4/detection_loss‚1.8218412101268768 epoch‚7hö+?]Õ"× 1loss/SRqqt4/detection_loss‚1.8218412101268768 epoch‚7 2loss/SRqqt4/assignment_loss‚1.5773876555786843 0loss/FBt1/assignment_loss‚0.5850794333604427 /loss/FBt4/assignment_loss‚0.199735587764917 /loss/FBt1/detection_loss‚2.9883818924427032 /loss/FBt4/detection_loss‚3.2217600643634796 0loss/FBt2/assignment_loss‚0.4846670716917288 /loss/FRt1/assignment_loss‚4.273233210339265 /loss/FRt2/assignment_loss‚2.196493243177732 /loss/FRt2/detection_loss‚2.1190173445890346 0loss/FRt3/assignment_loss‚2.0850328219180203 .loss/FRt1/detection_loss‚2.782246933263891 /loss/FBt2/detection_loss‚3.3307902216911316 $ _timestamp‚1.788414215859538e+09 _runtime‚6826.271603385 0loss/FRt4/assignment_loss‚3.3292494817421985 1loss/SRqqt1/assignment_loss‚2.511211797595024 /loss/FRt4/detection_loss‚2.2503106582623262 /loss/FBt3/detection_loss‚0.7487716392834526 trainer/global_step‚9149 1loss/FBt3/assignment_loss‚0.19461381063221575 /loss/FRt3/detection_loss‚1.9572393267738575 /loss/SRqqt1/detection_loss‚2.46351569550378 1loss/SRqqt3/detection_loss‚1.2069374557584525 &loss/total_loss‚47.110281134062745 _step‚752 2loss/SRqqt2/assignment_loss‚1.5266957243935007 1loss/SRqqt3/assignment_loss‚1.652943461885055 1loss/SRqqt2/detection_loss‚1.6031253919249675ð¼kGiœÖ"‚(:‘ •’äÔùѳ™ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.196' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.839199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yg»ø.×"‚(:£ •’äÔ…Ù› disk.sda3.out‚ 269.71484375 disk.drbd1777.in‚0& disk.nvme0n1.out‚71864.28271484375 disk.dm-0.in‚ 711.06640625( disk.nvme0c0n1.out‚71864.28271484375 disk.sda2.in‚0 disk.sda.out‚ 269.71484375 disk.rbd0.in‚0 cpu‚25.372435214634848 network.sent‚ 2.1761371e+07 network.recv‚ 2.471214e+06% disk./.usageGB‚1097.8607711791992 disk.sda1.in‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 711.06640625 disk.sda3.in‚0 disk.nvme0n1.in‚ 711.06640625 disk.dm-1.in‚0 disk.dm-1.out‚ 269.71484375$ proc.memory.availableMB‚248.3125 disk.sda2.out‚0# disk.dm-0.out‚71864.28271484375 disk.rbd0.out‚0$ memory_percent‚98.48442077636719# proc.memory.rssMB‚ 2879.48828125 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 3.7421875 disk.rbd2.in‚0 disk.sda.in‚0) proc.memory.percent‚17.57500171661377 proc.cpu.threads‚17) disk./.usagePercent‚73.67835752037902 disk.rbd1.in‚0! disk.rbd2.out‚157.05712890625 disk.rbd3.out‚0µã\ Ø"‚(:… ¤’äÔÖ§´™ gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚92.223' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.4446 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆŠ %Ù"‚(:š ¤’äÔÚêâ›$ disk./.usageGB‚1097.860450744629 disk.rbd1.in‚0 disk.rbd1.out‚ 3.7421875 disk.rbd0.out‚0 disk.rbd2.in‚0$ memory_percent‚98.48299026489258 network.sent‚ 2.1871691e+07) disk./.usagePercent‚73.67833601574614% disk.nvme0c0n1.out‚71963.76953125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.out‚71963.76953125 disk.rbd2.out‚157.1591796875" disk.nvme0c0n1.in‚ 711.06640625 disk.sda3.out‚ 270.37890625& proc.memory.availableMB‚ 248.546875) proc.memory.percent‚17.57500171661377 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2879.48828125 disk.nvme0n1.in‚ 711.06640625 disk.dm-0.in‚ 711.06640625 disk.sda.in‚0 disk.drbd1777.in‚0# disk.nvme0n1.out‚71963.76953125 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.sda.out‚ 270.37890625 disk.drbd1777.out‚0 cpu‚25.371731595187267 network.recv‚ 2.477112e+06 disk.dm-1.out‚ 270.37890625 disk.rbd0.in‚0 proc.cpu.threads‚17ÅRÎ2Ú"- _wandbruntime‚6860  _runtime‚6860[:6ŽÉÛ"à _runtime‚6860.562944524 _step‚753 ' lr-AdamW/pg1‚1.6599920836820493e-05 ' lr-AdamW/pg2‚1.6599920836820493e-05 trainer/global_step‚9199 % _timestamp‚1.7884142501507528e+09n@&}ÎÜ"È ' lr-AdamW/pg2‚1.6599920836820493e-05 trainer/global_step‚9199 % _timestamp‚1.7884142501507528e+09 _runtime‚6860.562944524 _step‚753 ' lr-AdamW/pg1‚1.6599920836820493e-05ñ>“rÔ2Ý"- _wandbruntime‚6860  _runtime‚6860Æp¦Þ"çO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 % _timestamp‚1.7884142501515586e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚259210365712384 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚259210347709440 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121195557 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121195557 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚272183411162624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 122868258 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚259517398975850 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272183224159744 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚274655509115904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚246581739621376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚272183224159744 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚259210365712384 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12973045450240 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307141229 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12935494233070 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185945670 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70743106 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12972876450304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚272183411162624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52125154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚287852784801792 _runtime‚6860.563852465 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12973045450240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚287853019326464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 122868786 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12972876450304 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 185925632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚246581722582016 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 185945670 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185925632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚526 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚274655485175808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52125680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13197299625984 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121195559 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70743104 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13197510210560 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12935659354474 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121195559 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70743104 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13197510210560 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12935659354474 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272183224159744 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚274655509115904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚246581739621376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚272183224159744 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚259210365712384 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12973045450240 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307141229 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12935494233070 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185945670 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70743106 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12972876450304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚272183411162624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52125154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚287852784801792 _runtime‚6860.563852465 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12973045450240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚287853019326464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 122868786 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12972876450304 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 185925632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚246581722582016 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 185945670 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185925632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚526 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚274655485175808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52125680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13197299625984 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 185945419 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚259544731322938 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚259237617074176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307173701 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121208216 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚274684422064128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121208218 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚259237635077120 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚246607665764352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 307153635 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚6861.218460315 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 _step‚755 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 185965483 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12974269588480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70750483 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 185945419 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52131038 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185965483 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122881009 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚259544549161966 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52130528 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12936883397614 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121208218 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12974438601728 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚246607682803712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70750481 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 307153635 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121208216 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚272212073678848 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12937048519226 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚274684398124032 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13198934001152 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 307173701 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12974438601728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚287883125748224 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚272211886662656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12974269588480 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122881521 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0£…në%â"öM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122881009 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚259544549161966 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52130528 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12936883397614 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121208218 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12974438601728 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚246607682803712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70750481 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 307153635 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121208216 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚272212073678848 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12937048519226 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚274684398124032 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13198934001152 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 307173701 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12974438601728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚287883125748224 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚272211886662656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12974269588480 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122881521 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚9199 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚272212073678848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 185945419 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚259544731322938 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚259237617074176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307173701 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121208216 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚274684422064128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121208218 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚259237635077120 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚246607665764352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 307153635 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚6861.218460315 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 _step‚755 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 185965483 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12974269588480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70750483 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 185945419 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52131038 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_trai7o´¤n_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185965483 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064óÞyv€2ã"- _wandbruntime‚6861  _runtime‚686155¡•Xä"Ò 0loss/FRt3/assignment_loss‚2.7213777914279844 0loss/SRqqt3/detection_loss‚1.622235140691583 _runtime‚6861.219639066 trainer/global_step‚9199 .loss/FRt1/assignment_loss‚4.48065533357508 0loss/FRt2/assignment_loss‚2.4942967210497176 0loss/FRt4/assignment_loss‚2.8703063600941707 /loss/FRt3/detection_loss‚2.2180769098241155 .loss/FRt4/detection_loss‚2.005458854792411 1loss/SRqqt1/detection_loss‚3.0413272369790962 2loss/SRqqt3/assignment_loss‚1.4454394802451134 2loss/SRqqt4/assignment_loss‚1.6208574149125028 0loss/FBt2/assignment_loss‚0.3473520986735821 /loss/FRt2/detection_loss‚2.0976747133369957 /loss/FBt3/detection_loss‚0.6599986365548418 epoch‚7 1loss/SRqqt1/assignment_loss‚2.249834371937646 /loss/FRt1/detection_loss‚2.7158919748137977 1loss/SRqqt4/detection_loss‚1.6092175118765741 0loss/FBt1/assignment_loss‚0.3634743405620611 1loss/FBt3/assignment_loss‚0.14472680580005415 0loss/FBt4/assignment_loss‚0.4719608399289947 % _timestamp‚1.7884142508075929e+09 _step‚756 /loss/FBt2/detection_loss‚3.7404624819755554 ,loss/FBt4/detection_loss‚2.5319962565388 %loss/total_loss‚48.10559155350223 2loss/SRqqt2/assignment_loss‚1.4305812134341875 /loss/FBt1/detection_loss‚3.5655061423778536 1loss/SRqqt2/detection_loss‚1.6568829220995096”¿æ]å"× /loss/FRt1/detection_loss‚2.7158919748137977 1loss/SRqqt4/detection_loss‚1.6092175118765741 0loss/FBt1/assignment_loss‚0.3634743405620611 1loss/FBt3/assignment_loss‚0.14472680580005415 0loss/FBt4/assignment_loss‚0.4719608399289947 % _timestamp‚1.7884142508075929e+09 _step‚756 /loss/FBt2/detection_loss‚3.7404624819755554 ,loss/FBt4/detection_loss‚2.5319962565388 %loss/total_loss‚48.10559155350223 2loss/SRqqt2/assignment_loss‚1.4305812134341875 /loss/FBt1/detection_loss‚3.5655061423778536 1loss/SRqqt2/detection_loss‚1.6568829220995096 0loss/FRt3/assignment_loss‚2.7213777914279844 0loss/SRqqt3/detection_loss‚1.622235140691583 _runtime‚6861.219639066 trainer/global_step‚9199 .loss/FRt1/assignment_loss‚4.48065533357508 0loss/FRt2/assignment_loss‚2.4942967210497176 0loss/FRt4/assignment_loss‚2.8703063600941707 /loss/FRt3/detection_loss‚2.2180769098241155 .loss/FRt4/detection_loss‚2.005458854792411 1loss/SRqqt1/detection_loss‚3.0413272369790962 2loss/SRqqt3/assignment_loss‚1.4454394802451134 2loss/SRqqt4/assignment_loss‚1.6208574149125028 0loss/FBt2/assignment_loss‚0.3473520986735821 /loss/FRt2/detection_loss‚2.0976747133369957 /loss/FBt3/detection_loss‚0.6599986365548418 epoch‚7 1loss/SRqqt1/assignment_loss‚2.249834371937646ô°4‘æ"‚(:† ³’äÔª‚³™ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.232' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.6464 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«7#ç"‚(:˜ ³’äÔ܈à› disk.dm-1.in‚0 network.sent‚ 2.1875048e+07) disk./.usagePercent‚73.59617449375226% disk./.usageGB‚1096.6361846923828 disk.rbd2.out‚158.7333984375 disk.sda2.in‚0 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.dm-1.out‚ 271.0546875 disk.rbd3.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚ 711.06640625$ memory_percent‚98.48337173461914& proc.memory.availableMB‚ 248.484375% disk.nvme0c0n1.out‚72062.94921875 disk.sda3.out‚ 271.0546875# disk.nvme0n1.out‚72062.94921875 disk.rbd3.out‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0) proc.memory.percent‚17.57500171661377 proc.cpu.threads‚17 network.recv‚ 2.480776e+06 disk.sda3.in‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 711.06640625 disk.sda.out‚ 271.0546875 disk.rbd1.out‚ 3.7421875# proc.memory.rssMB‚ 2879.48828125 cpu‚25.370995533862786 disk.rbd2.in‚0 disk.dm-0.in‚ 711.06640625 disk.drbd1777.in‚0 disk.dm-0.out‚72062.94921875ºÊ’mè"‚(:… Â’äÔ„Š¡™ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚88.957' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7914 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dÑ(3'é"‚(:œ Â’äÔÀã°›) disk./.usagePercent‚73.60709244706501 disk.dm-0.in‚ 711.06640625 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 711.06640625 disk.sda2.in‚0 disk.rbd1.out‚ 3.7421875 disk.nvme0n1.in‚ 711.06640625 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 271.421875 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 271.421875 disk.rbd0.in‚0' proc.memory.availableMB‚ 247.9453125) proc.memory.percent‚17.57500171661377( disk.nvme0c0n1.out‚72230.08642578125 disk.sda3.out‚ 271.421875 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.rbd2.out‚158.8115234375 disk.rbd3.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚72230.08642578125 network.sent‚ 2.198403e+07# disk./.usageGB‚1096.79887008667 disk.drbd1777.out‚0$ memory_percent‚98.48666191101074# proc.memory.rssMB‚ 2879.48828125 network.recv‚ 2.487568e+06# disk.dm-0.out‚72230.08642578125 disk.rbd2.in‚0 cpu‚25.370301564488148r‡Ä,2ê"- _wandbruntime‚6895  _runtime‚6895ŽU^¿Èë" ' lr-AdamW/pg1‚1.6063012044470342e-05 ' lr-AdamW/pg2‚1.6063012044470342e-05 trainer/global_step‚9249 % _timestamp‚1.7884142847731724e+09 _runtime‚ 6895.18533972 _step‚7575'Íì"Ç ' lr-AdamW/pg1‚1.6063012044470342e-05 ' lr-AdamW/pg2‚1.6063012044470342e-05 trainer/global_step‚9249 % _timestamp‚1.7884142847731724e+09 _runtime‚ 6895.18533972 _step‚757õ‰F®62í"- _wandbruntime‚6895  _runtime‚6895´%sJí'î"çO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121828507 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 186918334 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚273616363719168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 186938372 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 308746841 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13005131172882 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121828509 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 % _timestamp‚1.7884142847740016e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚537 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚276101132587008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186918334 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 308766881 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71111956 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 trainer/global_step‚9249 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13268220988928 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚273616363719168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚289369353575936 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚260573833949184 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 186918334 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚273616363719168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 186938372 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 308746841 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13005131172882 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121828509 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 % _timestamp‚1.7884142847740016e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚537 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚276101132587008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186918334 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 308766881 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71111956 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 trainer/global_step‚9249 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13268220988928 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚273616363719168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚289369353575936 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0övn]Ð2ð"- _wandbruntime‚6895  _runtime‚6895s%kš#ñ"ñM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚516 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 186958185 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52402388 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 186938121 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚247904824881152 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚260601103313920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚260911362258146 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚9249 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71119331 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 _runtime‚6895.839536976 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 186958185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚273645026222080 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121841166 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121841168 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 308779287 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 308799353 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚518 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚276130045535232 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 186938121 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13044109924352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13043940911104 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123521721 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚247904841920512 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13044109924352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13269640350720 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123521203 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13006355216022 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 308799353 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ÔA›Ô260601085310976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 308779287 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚273645213238272 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123521203 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13006355216022 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 308799353 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚260601085310976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 308779287 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚273645213238272 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 186958185 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52402388 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 186938121 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚247904824881152 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚260601103313920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚260911362258146 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚9249 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71119331 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 _runtime‚6895.839536976 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 186958185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚273645026222080 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121841166 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121841168 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 308779287 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 308799353 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚518 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚276130045535232 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 186938121 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13044109924352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13043940911104 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123521721 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚247904841920512 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13044109924352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13269640350720 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚249176057854976 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚290885940590592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13339160592384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187931074 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 124149666 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13112388095488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122461459 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52668862 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _step‚762 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13112388095488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 310372493 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 187931074 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚275049690281472 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13112219095552 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚262250660846266 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚277546803938304 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275049503278592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 310392533 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚515 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 $ _timestamp‚1.788414319689259e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122461457 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 124150183 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚277546779998208 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚249176057854976 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚290885940590592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13339160592384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187931074 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 124149666 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13112388095488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122461459 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52668862 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _step‚762 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13112388095488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 310372493 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 187931074 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚275049690281472 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13112219095552 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚262250660846266 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚277546803938304 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275049503278592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 310392533 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚515 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 $ _timestamp‚1.788414319689259e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122461457 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 124150183 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚277546779998208 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚261964571550720 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚262277993193354 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚275078352797696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13075827034430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚277575716886528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71488181 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 310425005 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122474118 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚249202001037312 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52674096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚261964553547776 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚261964553547776 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124162277 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 # _timestamp‚1.78841432033521e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187930823 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 310425005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52674617 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚277575692946432 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124162800 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚275078352797696 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122474118 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71488183 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13113612233728 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13340803223040 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚261964571550720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚275078165781504 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _runtime‚6930.747906906S<ÿÿú&„#ôM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚262277993193354 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚275078352797696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13075827034430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚277575716886528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71488181 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 310425005 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122474118 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚249202001037312 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52674096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚261964553547776 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚261964553547776 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124162277 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 # _timestamp‚1.78841432033521e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187930823 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 310425005 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52674617 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚277575692946432 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124162800 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚275078352797696 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122474118 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71488183 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13113612233728 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13340803223040 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚261964571550720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚275078165781504 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _runtime‚6930.747906906 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13075992156042 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13113612233728 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚290916520109568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚763 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 187950887 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 187950887 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚261964571550720 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257û!ÈI92…#- _wandbruntime‚6930  _runtime‚6930÷ÕN†#È 2loss/SRqqt3/assignment_loss‚1.3489296391606331 2loss/SRqqt4/assignment_loss‚1.1720007433677497 -loss/FRt4/detection_loss‚2.15639255060391 0loss/SRqqt3/detection_loss‚1.317763460551699 % _timestamp‚1.7884143203369093e+09 0loss/FRt3/assignment_loss‚2.4569840022811182 /loss/FRt3/detection_loss‚1.9842641292898744 %loss/total_loss‚50.96775012423252 1loss/FBt3/assignment_loss‚0.23688799836942945 0loss/FBt4/assignment_loss‚0.7293093287220332 _runtime‚ 6930.74892265 .loss/FRt2/assignment_loss‚3.13410885496573 0loss/SRqqt1/assignment_loss‚2.64670572357793 0loss/FBt2/assignment_loss‚0.6524487331553246 0loss/SRqqt1/detection_loss‚2.507616234402503 .loss/FBt2/detection_loss‚4.725374177098274 trainer/global_step‚9299 _step‚764 /loss/FBt1/assignment_loss‚0.436091113318576 0loss/SRqqt4/detection_loss‚2.134451675026313 -loss/FBt1/detection_loss‚3.06006430917316 1loss/SRqqt2/assignment_loss‚1.591376440883693 .loss/FBt4/detection_loss‚2.450437000164619 /loss/FRt1/assignment_loss‚4.777717051967498 1loss/SRqqt2/detection_loss‚1.7130927761732522 epoch‚7 0loss/FRt4/assignment_loss‚3.7558531192216007 /loss/FRt1/detection_loss‚2.8431235763334457 .loss/FRt2/detection_loss‚2.340954664755951 /loss/FBt3/detection_loss‚0.7958028216682076j’US‡#Í 2loss/SRqqt3/assignment_loss‚1.3489296391606331 2loss/SRqqt4/assignment_loss‚1.1720007433677497 -loss/FRt4/detection_loss‚2.15639255060391 0loss/SRqqt3/detection_loss‚1.317763460551699 % _timestamp‚1.7884143203369093e+09 0loss/FRt3/assignment_loss‚2.4569840022811182 /loss/FRt3/detection_loss‚1.9842641292898744 %loss/total_loss‚50.96775012423252 1loss/FBt3/assignment_loss‚0.23688799836942945 0loss/FBt4/assignment_loss‚0.7293093287220332 _runtime‚ 6930.74892265 .loss/FRt2/assignment_loss‚3.13410885496573 0loss/SRqqt1/assignment_loss‚2.64670572357793 0loss/FBt2/assignment_loss‚0.6524487331553246 0loss/SRqqt1/detection_loss‚2.507616234402503 .loss/FBt2/detection_loss‚4.725374177098274 trainer/global_step‚9299 _step‚764 /loss/FBt1/assignment_loss‚0.436091113318576 0loss/SRqqt4/detection_loss‚2.134451675026313 -loss/FBt1/detection_loss‚3.06006430917316 1loss/SRqqt2/assignment_loss‚1.591376440883693 .loss/FBt4/detection_loss‚2.450437000164619 /loss/FRt1/assignment_loss‚4.777717051967498 1loss/SRqqt2/detection_loss‚1.7130927761732522 epoch‚7 0loss/FRt4/assignment_loss‚3.7558531192216007 /loss/FRt1/detection_loss‚2.8431235763334457 .loss/FRt2/detection_loss‚2.340954664755951 /loss/FBt3/detection_loss‚0.7958028216682076ü¦¸¬ˆ#‚(:† þ’äÔ”öž™ gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAîsOçllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.942' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨8&ý/‰#‚(:¤ þ’äÔš×Õ› proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda3.in‚0 disk.rbd1.out‚ 3.7421875( proc.memory.availableMB‚ 248.15234375$ disk./.usageGB‚1096.799129486084 disk.dm-1.out‚ 273.66015625 disk.rbd0.in‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2879.48828125) proc.memory.percent‚17.57500171661377) disk./.usagePercent‚73.60710985557735 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.out‚0& disk.nvme0n1.out‚72232.12451171875# memory_percent‚98.4853982925415! disk.rbd2.out‚158.99169921875 disk.rbd2.in‚0 disk.sda.out‚ 273.66015625 network.sent‚ 2.2108981e+07 disk.sda2.out‚0 disk.dm-0.in‚ 711.06640625 network.recv‚ 2.505759e+06 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 711.06640625( disk.nvme0c0n1.out‚72232.12451171875 disk.sda3.out‚ 273.66015625 disk.sda.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 711.06640625 disk.drbd1777.in‚0# disk.dm-0.out‚72232.12451171875 cpu‚25.36750606501276”SÔ¤œŠ#‚(:‘ “äÔšâ«™ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚104.209' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.841800000000003 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aÊB×-‹#‚(:¢ “äÔ¯¸Èœ network.sent‚ 2.2217721e+07 network.recv‚ 2.511988e+06 disk.sda.in‚0 disk.sda1.in‚0& disk.nvme0n1.out‚72232.51123046875 disk.rbd0.out‚0 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-1.out‚ 274.36328125" disk.nvme0c0n1.in‚ 711.06640625# proc.memory.rssMB‚ 2879.48828125) disk./.usagePercent‚73.60711113561501% disk./.usageGB‚1096.7991485595703 disk.sda2.out‚0 disk.rbd1.in‚0# disk.dm-0.out‚72232.51123046875 disk.rbd3.out‚0( disk.nvme0c0n1.out‚72232.51123046875 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 274.36328125 disk.drbd1777.in‚0 disk.dm-0.in‚ 711.06640625 disk.rbd2.in‚0 cpu‚25.36685330461091 disk.sda3.in‚0 disk.rbd1.out‚ 3.7421875 disk.nvme0n1.in‚ 711.06640625 disk.dm-1.in‚0 disk.sda.out‚ 274.36328125! disk.rbd2.out‚160.56591796875# memory_percent‚98.4842300415039 disk.drbd1777.out‚0% proc.memory.availableMB‚ 248.34375) proc.memory.percent‚17.57500171661377ÜÝB¼2Œ#- _wandbruntime‚6965  _runtime‚6965Aöï6É#à ' lr-AdamW/pg2‚1.5010845575052227e-05 trainer/global_step‚9349 % _timestamp‚1.7884143551011107e+09 _runtime‚6965.513284132 _step‚765 ' lr-AdamW/pg1‚1.5010845575052227e-05k¹iÎŽ#È ' lr-AdamW/pg1‚1.5010845575052227e-05 ' lr-AdamW/pg2‚1.5010845575052227e-05 trainer/global_step‚9349 % _timestamp‚1.7884143551011107e+09 _runtime‚6965.513284132 _step‚765ý¨ªÛö2#- _wandbruntime‚6965  _runtime‚6965W`†í'#çO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 312018185 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71849656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71849654 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 124788479 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚292402513002496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚250473199932416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚9349 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚529 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚276482642838016 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚263300752419840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚527 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚263617109620710 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚276482829840896 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 124787950 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _step‚766 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚276482642838016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚263300770422784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123094407 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 188923776 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13410085593088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884143551019025e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13181890418176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚263300770422784 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 188903738 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚263617291781474 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13181890418176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123094407 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13182059418112 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚278992427409408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚278992451349504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123094409 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚276482829840896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚292402747527168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188923776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 188903738 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13182059418112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52938823 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13144074809698 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13143909688294 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚6965.514273296 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 311998145 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚263300752419840 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13410296177664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚278992451349504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123094409 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚276482829840896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚292402747527168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188923776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 188903738 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13182059418112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52938823 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13144074809698 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13143909688294 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚6965.514273296 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 311998145 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚263300752419840 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13410296177664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚276482642838016 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚263300752419840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚527 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚263617109620710 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚276482829840896 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 124787950 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _step‚766 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚276482642838016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚263300770422784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123094407 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 188923776 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13410085593088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884143551019025e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13181890418176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚263300770422784 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 188903738 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚263617291781474þ'žˆ&2’#- _wandbruntime‚6966  _runtime‚6966‰x €–"“#ñM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚263328039787520 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚263328039787520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 188943589 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188923525 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 trainer/global_step‚9349 _runtime‚6966.217099084 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚263644624128562 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123107068 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13145298852838 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123107068 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124800770 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚279021340357632 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚250499143114752 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71857031 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 312050657 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188923525 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚540 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52943739 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13411712777728 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 % _timestamp‚1.7884143558042536e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13183452569600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚263644441967590 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13183283556352 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123107066 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124801310 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13411502206464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188943589 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312050657 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13145463974450 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚263328021784576 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248&•…4ü&”#öM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188943589 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312050657 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13145463974450 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚263328021784576 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚263328039787520 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚263328039787520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 188943589 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188923525 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 trainer/global_step‚9349 _runtime‚6966.217099084 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚263644624128562 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123107068 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13145298852838 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123107068 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124800770 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚279021340357632 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚250499143114752 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71857031 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 312050657 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188923525 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚540 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52943739 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13411712777728 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 % _timestamp‚1.7884143558042536e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13183452569600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚263644441967590 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13183283556352 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123107066 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124801310 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13411502206464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944ÿÜ_â<2•#- _wandbruntime‚6966  _runtime‚6966é˜ê\–#Ö 0loss/FBt1/assignment_loss‚0.4715508075496473 1loss/FBt3/assignment_loss‚0.19125267682800873 .loss/FRt1/detection_loss‚2.988938597838084 .loss/FRt4/detection_loss‚2.168164804875851 1loss/SRqqt2/detection_loss‚1.4391532490986425 0loss/SRqqt4/detection_loss‚2.127372396263209 0loss/FRt2/assignment_loss‚2.6625749003887176 %loss/total_loss‚48.76373905877506 0loss/FRt4/assignment_loss‚3.3433972084522248 2loss/SRqqt2/assignment_loss‚1.2791733352996006 2loss/SRqqt3/assignment_loss‚1.2816763181450233 1loss/SRqqt3/detection_loss‚1.5945837029589798 /loss/FRt2/detection_loss‚2.2162136754393575 /loss/FRt3/detection_loss‚1.8610401648962707 epoch‚7 _step‚768 % _timestamp‚1.7884143558061233e+09 /loss/FRt1/assignment_loss‚5.067375516891479 0loss/FRt3/assignment_loss‚3.0599350772405924 2loss/SRqqt4/assignment_loss‚1.8381133990531617 1loss/FBt4/assignment_loss‚0.40339638428905833 0loss/SRqqt1/detection_loss‚2.350748588357653 /loss/FBt2/detection_loss‚3.2959047158559165 .loss/FBt3/detection_loss‚0.764711817959477 2loss/SRqqt1/assignment_loss‚2.6525090081351146 1loss/FBt2/assignment_loss‚0.35559846460819244 /loss/FBt1/detection_loss‚3.3133884838649204 _runtime‚6966.218162767 /loss/FBt4/detection_loss‚2.0369657644858727 trainer/global_step‚9349yÙµKa—#Û 0loss/FRt4/assignment_loss‚3.3433972084522248 2loss/SRqqt2/assignment_loss‚1.2791733352996006 2loss/SRqqt3/assignment_loss‚1.2816763181450233 1loss/SRqqt3/detection_loss‚1.5945837029589798 /loss/FRt2/detection_loss‚2.2162136754393575 /loss/FRt3/detection_loss‚1.8610401648962707 epoch‚7 _step‚768 % _timestamp‚1.7884143558061233e+09 /loss/FRt1/assignment_loss‚5.067375516891479 0loss/FRt3/assignment_loss‚3.0599350772405924 2loss/SRqqt4/assignment_loss‚1.8381133990531617 1loss/FBt4/assignment_loss‚0.40339638428905833 0loss/SRqqt1/detection_loss‚2.350748588357653 /loss/FBt2/detection_loss‚3.2959047158559165 .loss/FBt3/detection_loss‚0.764711817959477 2loss/SRqqt1/assignment_loss‚2.6525090081351146 1loss/FBt2/assignment_loss‚0.35559846460819244 /loss/FBt1/detection_loss‚3.3133884838649204 _runtime‚6966.218162767 /loss/FBt4/detection_loss‚2.0369657644858727 trainer/global_step‚9349 0loss/FBt1/assignment_loss‚0.4715508075496473 1loss/FBt3/assignment_loss‚0.19125267682800873 .loss/FRt1/detection_loss‚2.988938597838084 .loss/FRt4/detection_loss‚2.168164804875851 1loss/SRqqt2/detection_loss‚1.4391532490986425 0loss/SRqqt4/detection_loss‚2.127372396263209 0loss/FRt2/assignment_loss‚2.6625749003887176 %loss/total_loss‚48.76373905877506€“y‘˜#‚(:† œ“äÔ¹žÀš gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0` 7/™#‚(:¤ œ“äÔøßðœ! disk.rbd2.out‚160.70654296875 disk.rbd3.out‚0 disk.sda2.out‚0 disk.drbd1777.out‚0( proc.memory.availableMB‚ 248.55859375 network.recv‚ 2.515874e+06) disk./.usagePercent‚73.60711113561501 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 711.06640625$ memory_percent‚98.48291873931885 cpu‚25.366171893493252 disk.sda3.out‚ 275.1171875 disk.drbd1777.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚72232.92822265625 network.sent‚ 2.2223049e+07 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda.out‚ 275.1171875 disk.sda1.in‚0 disk.sda1.out‚0% disk./.usageGB‚1096.7991485595703 disk.dm-1.out‚ 275.1171875 disk.rbd0.in‚0 disk.sda3.in‚0# disk.dm-0.out‚72232.92822265625# proc.memory.rssMB‚ 2879.48828125" disk.nvme0c0n1.in‚ 711.06640625 disk.rbd1.out‚ 3.7421875) proc.memory.percent‚17.57500171661377( disk.nvme0c0n1.out‚72232.92822265625 disk.nvme0n1.in‚ 711.06640625Ä‹x‘š#‚(:† «“äÔû陵 gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚120.392' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0784 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=ri£!›#‚(:– «“äÔ ùž disk.sda.out‚275.6875 disk.sda1.in‚0 disk.drbd1777.out‚0# disk.dm-0.out‚72234.95947265625 disk.dm-1.out‚275.6875" disk.nvme0c0n1.in‚ 711.06640625 disk.rbd3.in‚0 network.sent‚ 2.233151e+07 disk.rbd0.out‚0# memory_percent‚98.4837532043457& proc.memory.availableMB‚ 248.421875) proc.memory.percent‚17.57500171661377 cpu‚25.36549300323463 proc.cpu.threads‚17 disk.sda.in‚0 disk.rbd2.in‚0% disk./.usageGB‚1096.6300315856934 disk.drbd1777.in‚0 disk.rbd1.in‚0 network.recv‚ 2.521438e+06 disk.sda3.in‚0 disk.rbd1.out‚ 3.7421875& disk.nvme0n1.out‚72234.95947265625 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚72234.95947265625# proc.memory.rssMB‚ 2879.48828125 disk.sda2.in‚0 disk.nvme0n1.in‚ 711.06640625 disk.dm-0.in‚ 711.06640625 disk.rbd0.in‚0! disk.rbd2.out‚160.70654296875) disk./.usagePercent‚73.59576155359953 disk.sda3.out‚275.6875s]2œ#- _wandbruntime‚7000  _runtime‚7000«fDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚277915782397440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚519 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚264664238659584 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚251770359049216 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13213381506702 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13251561740800 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚264664220656640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚264664220656640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13251561740800 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚521 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 _runtime‚ 7000.51314392 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13251730740736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72218506 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123727359 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53209703 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚280438098760704 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚277915969400320 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53209184 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 trainer/global_step‚9399 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 189896440 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125427688 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚293919022770176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13213546628106 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72218504 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚264664238659584 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13481160631296 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 189916478 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728À-8ö¡#ëO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚277915782397440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚519 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚264664238659584 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚251770359049216 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13213381506702 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13251561740800 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚264664220656640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚264664220656640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13251561740800 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚521 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 _runtime‚ 7000.51314392 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13251730740736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72218506 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123727359 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53209703 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚280438098760704 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚277915969400320 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53209184 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 trainer/global_step‚9399 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 189896440 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125427688 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚293919022770176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13213546628106 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72218504 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚264664238659584 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13481160631296 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 189916478 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 189916478 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚277915782397440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224‚o™ 2¢#- _wandbruntime‚7001  _runtime‚7001Wà]T÷&£#ñM ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13214935792858 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13252954878976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚500 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚9399 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚264691508024320 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251796302231552 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125440870 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚293949587990016 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚265011072902798 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 189936291 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13482365709824 _runtime‚7001.178994484 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13252954878976 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 189936291 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 189916227 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚277944444900352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 125441372 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 189916227 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123740016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72225883 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53214989 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 313656243 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚277944631916544 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13214770671246 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚502 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 313676309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 313656243 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13253123892224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚264691490021376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚265011255063770 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123740016 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313676309 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚293949353478656 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13482576281088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚264691490021376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚771a\Åü&¤#öM _step‚771 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13214935792858 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13252954878976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚500 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚9399 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚264691508024320 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251796302231552 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125440870 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚293949587990016 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚265011072902798 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 189936291 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13482365709824 _runtime‚7001.178994484 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13252954878976 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 189936291 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 189916227 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚277944444900352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 125441372 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 189916227 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123740016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72225883 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53214989 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 313656243 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚277944631916544 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13214770671246 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚502 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 313676309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 313656243 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13253123892224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚264691490021376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚265011255063770 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123740016 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313676309 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚293949353478656 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13482576281088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚264691490021376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0ƒ”Xa2¥#- _wandbruntime‚7001  _runtime‚7001†OãU¦#Ï 1loss/SRqqt4/detection_loss‚1.6935255420657824 epoch‚7 /loss/FRt1/assignment_loss‚4.797032117843628 0loss/FRt3/assignment_loss‚1.9193427165349324 1loss/FBt3/assignment_loss‚0.11259919275234627 0loss/SRqqt2/detection_loss‚1.368801251752302 /loss/FBt3/detection_loss‚1.2484184796766689 % _timestamp‚1.7884143907681484e+09 2loss/SRqqt1/assignment_loss‚2.2359641763296993 /loss/FBt1/assignment_loss‚0.735475817862747 .loss/FRt1/detection_loss‚2.955450021168765 /loss/FRt2/detection_loss‚2.3873211951418356 /loss/FRt4/detection_loss‚2.0907235783338547 0loss/SRqqt1/detection_loss‚2.497868080933889 .loss/FBt4/detection_loss‚2.923853199928999 1loss/FBt2/assignment_loss‚0.15124345748343634 0loss/FBt4/assignment_loss‚0.6537644443778845 0loss/SRqqt3/detection_loss‚1.038423659824408 .loss/FBt1/detection_loss‚2.839169149215405 /loss/FBt2/detection_loss‚2.7694213887055716 _runtime‚7001.180498738 0loss/FRt2/assignment_loss‚3.2398931980133057 1loss/SRqqt3/assignment_loss‚1.296224443299266 trainer/global_step‚9399 /loss/FRt4/assignment_loss‚3.674832494258881 2loss/SRqqt4/assignment_loss‚1.4266675156804751 /loss/FRt3/detection_loss‚1.9603960251106936 $loss/total_loss‚47.4174383560256 _step‚772 1loss/SRqqt2/assignment_loss‚1.401027209730819,XZ§#Ô .loss/FBt1/detection_loss‚2.839169149215405 /loss/FBt2/detection_loss‚2.7694213887055716 _runtime‚7001.180498738 0loss/FRt2/assignment_loss‚3.2398931980133057 1loss/SRqqt3/assignment_loss‚1.296224443299266 trainer/global_step‚9399 /loss/FRt4/assignment_loss‚3.674832494258881 2loss/SRqqt4/assignment_loss‚1.4266675156804751 /loss/FRt3/detection_loss‚1.9603960251106936 $loss/total_loss‚47.4174383560256 _step‚772 1loss/SRqqt2/assignment_loss‚1.401027209730819 1loss/SRqqt4/detection_loss‚1.6935255420657824 epoch‚7 /loss/FRt1/assignment_loss‚4.797032117843628 0loss/FRt3/assignment_loss‚1.9193427165349324 1loss/FBt3/assignment_loss‚0.11259919275234627 0loss/SRqqt2/detection_loss‚1.368801251752302 /loss/FBt3/detection_loss‚1.2484184796766689 % _timestamp‚1.7884143907681484e+09 2loss/SRqqt1/assignment_loss‚2.2359641763296993 /loss/FBt1/assignment_loss‚0.735475817862747 .loss/FRt1/detection_loss‚2.955450021168765 /loss/FRt2/detection_loss‚2.3873211951418356 /loss/FRt4/detection_loss‚2.0907235783338547 0loss/SRqqt1/detection_loss‚2.497868080933889 .loss/FBt4/detection_loss‚2.923853199928999 1loss/FBt2/assignment_loss‚0.15124345748343634 0loss/FBt4/assignment_loss‚0.6537644443778845 0loss/SRqqt3/detection_loss‚1.038423659824408„êÄ{®œ¨#‚(:‘ º“äÔÑí¿™ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚122.865' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.572999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}Ž#©#‚(:˜ º“äÔÂþ’œ disk.sda3.in‚0$ memory_percent‚98.48506450653076 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0 network.recv‚ 2.526262e+06 disk.dm-0.in‚ 711.06640625" disk.dm-0.out‚72235.6396484375 disk.sda.out‚ 276.171875 disk.dm-1.out‚ 276.171875 disk.sda3.out‚ 276.171875) proc.memory.percent‚17.57500171661377 cpu‚25.36479242138525 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 248.20703125 disk.rbd2.out‚ 160.87109375 disk.nvme0n1.in‚ 711.06640625 disk.rbd1.out‚ 3.7421875( disk./.usagePercent‚73.5956483982694 disk.rbd0.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda.in‚0% disk.nvme0n1.out‚72235.6396484375 disk.rbd3.in‚0$ disk./.usageGB‚1096.628345489502 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 711.06640625 disk.rbd1.in‚0# proc.memory.rssMB‚ 2879.48828125 network.sent‚ 2.2337627e+07 disk.dm-1.in‚0' disk.nvme0c0n1.out‚72235.6396484375ǡ雪#‚(: É“äÔëÅ™ gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚99.821' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.964199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›µŒ¥&«#‚(:› É“äÔù­Â› disk.sda1.out‚0% disk.nvme0n1.out‚72369.2255859375 disk.dm-1.in‚0& proc.memory.availableMB‚ 247.546875 proc.cpu.threads‚17% disk./.usageGB‚1096.7584609985352 disk.sda2.out‚0 disk.sda3.out‚ 276.9921875 disk.dm-1.out‚ 276.9921875) proc.memory.percent‚17.57500171661377 disk.rbd2.out‚ 160.87890625 disk.sda.in‚0 disk.sda.out‚ 276.9921875 disk.rbd1.out‚ 3.7421875 disk.nvme0n1.in‚ 711.06640625$ memory_percent‚98.48909378051758 network.sent‚ 2.2446388e+07 disk.rbd3.in‚0 disk.rbd3.out‚0" disk.dm-0.out‚72369.2255859375 disk.drbd1777.out‚0 disk.dm-0.in‚ 711.06640625 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 711.06640625 cpu‚25.364091420301953( disk./.usagePercent‚73.6043805592548 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚72369.2255859375# proc.memory.rssMB‚ 2879.48828125 network.recv‚ 2.532682e+06ôÎÝJ¬#‚(:… Ø“äÔȉœ™ gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S³“%­#‚(:š Ø“äÔ÷ÏÄœ disk.sda2.out‚0 disk.sda3.out‚ 277.45703125$ disk.nvme0n1.out‚72370.376953125( proc.memory.availableMB‚ 248.03515625 cpu‚25.36345187739885 disk.rbd2.out‚ 162.53125 disk.sda2.in‚0 disk.sda.out‚ 277.45703125 disk.sda1.in‚0 network.sent‚ 2.2451711e+07% disk./.usageGB‚1096.8890533447266 disk.rbd3.in‚0 disk.drbd1777.in‚0) proc.memory.percent‚17.57500171661377 disk.dm-0.in‚ 711.06640625 disk.dm-1.out‚ 277.45703125 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda.in‚0 disk.drbd1777.out‚0$ memory_percent‚98.48611354827881 proc.cpu.threads‚17! disk.dm-0.out‚72370.376953125# proc.memory.rssMB‚ 2879.48828125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0 disk.rbd1.in‚0) disk./.usagePercent‚73.61314472118198" disk.nvme0c0n1.in‚ 711.06640625 disk.sda1.out‚0 disk.nvme0n1.in‚ 711.06640625& disk.nvme0c0n1.out‚72370.376953125 disk.rbd1.out‚ 3.7421875 network.recv‚ 2.536396e+06ž¼¬á2®#- _wandbruntime‚7035  _runtime‚7035Á)`ɯ#à _step‚773 ' lr-AdamW/pg1‚1.3988275604109244e-05 ' lr-AdamW/pg2‚1.3988275604109244e-05 trainer/global_step‚9449 % _timestamp‚1.7884144253854125e+09 _runtime‚7035.797590477 7ºí_°#È ' lr-AdamW/pg1‚1.3988275604109244e-05 ' lr-AdamW/pg2‚1.3988275604109244e-05 traC]¢ oiner/global_step‚9449 % _timestamp‚1.7884144253854125e+09 _runtime‚7035.797590477 _step‚773…ÌžÂ2±#- _wandbruntime‚7035  _runtime‚7035¿¨ê²ì'²#æO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 315269489 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 126067597 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72587356 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚279349108959744 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72587354 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚295435559531008 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚279348921956864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 190909180 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279348921956864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 190889142 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53480791 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221070336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13321233063424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13282853325110 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚266027688893440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚253067518166016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚266027706896384 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚266350553651890 _step‚774 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 190909180 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13552058369536 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚266350371491126 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124360309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124360307 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚281883746171904 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚253067535205376 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 $ _timestamp‚1.788414425386224e+09H ¥†ñ'³#ëO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245010432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13321402063360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124360307 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13321402063360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124360309 trainer/global_step‚9449 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚266027706896384 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 190889142 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13551837299200 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13283018446514 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚266027688893440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 190909180 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13552058369536 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚266350371491126 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124360309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124360307 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚281883746171904 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚253067535205376 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 $ _timestamp‚1.788414425386224e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 315269489 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 126067597 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72587356 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚279349108959744 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72587354 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚295435559531008 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚279348921956864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 190909180 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279348921956864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 190889142 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53480791 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221070336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13321233063424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13282853325110 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚266027688893440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚253067518166016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚266027706896384 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚266350553651890 _step‚774 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 315281895 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315301961 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 190908929 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚295465890570240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13322795214848 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13553255390208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 315301961 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124372966 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 _step‚775 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 190908929 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126080380 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚266377703838006 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚266054976261120 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13284407611266 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 _runtime‚7036.517834842 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72594731 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚266054976261120 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13322626201600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚281912635180032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884144261051087e+09 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160”5D(¶#öM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53485649 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚279377771475968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315301961 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 190908929 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚295465890570240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13322795214848 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13553255390208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 315301961 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124372966 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 _step‚775 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 190908929 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126080380 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚266377703838006 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚266054976261120 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13284407611266 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 _runtime‚7036.517834842 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72594731 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚266054976261120 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13322626201600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚281912635180032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884144261051087e+09 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 126080917 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53486184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚279377584459776 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124372968 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295466122984448 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 315281895 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 F a2·#- _wandbruntime‚7036  _runtime‚7036 EÁ[¸#Õ 2loss/SRqqt3/assignment_loss‚1.2414601589468393 2loss/FBt2/assignment_loss‚0.055891817750004215 1loss/SRqqt4/detection_loss‚2.0112565254911465 1loss/FBt3/assignment_loss‚0.25043741172423023 /loss/SRqqt2/detection_loss‚2.27296713792852 /loss/FBt1/detection_loss‚3.3165963113307955 2loss/SRqqt4/assignment_loss‚1.6624497384467023 0loss/FBt4/assignment_loss‚0.2569899730151519 /loss/FRt2/detection_loss‚2.6726984289976268 /loss/FRt1/detection_loss‚2.8935320556163786 /loss/FRt4/detection_loss‚2.2078719256406134 1loss/SRqqt1/detection_loss‚2.8475467614505603 %loss/total_loss‚49.37625384564788 .loss/FRt2/assignment_loss‚3.59962869148988 0loss/FRt4/assignment_loss‚2.8745111759672772 /loss/FRt1/assignment_loss‚4.641582862536112 2loss/SRqqt2/assignment_loss‚1.1613140280048053 1loss/FBt1/assignment_loss‚0.42609670684014417 1loss/SRqqt1/assignment_loss‚2.135785583892594 /loss/FRt3/detection_loss‚1.7765746956380706 epoch‚7 % _timestamp‚1.7884144261068032e+09 1loss/SRqqt3/detection_loss‚1.3219717663797466 /loss/FBt2/detection_loss‚3.5132384995619454 /loss/FBt3/detection_loss‚0.6076597766622902 .loss/FBt4/detection_loss‚3.991228646039963 trainer/global_step‚9449 _runtime‚7036.518940064 _step‚776 /loss/FRt3/assignment_loss‚1.636963166296482b¿4©`¹#Ú 1loss/FBt3/assignment_loss‚0.25043741172423023 /loss/SRqqt2/detection_loss‚2.27296713792852 /loss/FBt1/detection_loss‚3.3165963113307955 2loss/SRqqt4/assignment_loss‚1.6624497384467023 0loss/FBt4/assignment_loss‚0.2569899730151519 /loss/FRt2/detection_loss‚2.6726984289976268 /loss/FRt1/detection_loss‚2.8935320556163786 /loss/FRt4/detection_loss‚2.2078719256406134 1loss/SRqqt1/detection_loss‚2.8475467614505603 %loss/total_loss‚49.37625384564788 .loss/FRt2/assignment_loss‚3.59962869148988 0loss/FRt4/assignment_loss‚2.8745111759672772 /loss/FRt1/assignment_loss‚4.641582862536112 2loss/SRqqt2/assignment_loss‚1.1613140280048053 1loss/FBt1/assignment_loss‚0.42609670684014417 1loss/SRqqt1/assignment_loss‚2.135785583892594 /loss/FRt3/detection_loss‚1.7765746956380706 epoch‚7 % _timestamp‚1.7884144261068032e+09 1loss/SRqqt3/detection_loss‚1.3219717663797466 /loss/FBt2/detection_loss‚3.5132384995619454 /loss/FBt3/detection_loss‚0.6076597766622902 .loss/FBt4/detection_loss‚3.991228646039963 trainer/global_step‚9449 _runtime‚7036.518940064 _step‚776 /loss/FRt3/assignment_loss‚1.636963166296482 2loss/SRqqt3/assignment_loss‚1.2414601589468393 2loss/FBt2/assignment_loss‚0.055891817750004215 1loss/SRqqt4/detection_loss‚2.0112565254911465ˆü9º#‚(:… ç“äÔÅÔ«™ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚95.151' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.0302 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï~1Í"»#‚(:— ç“äÔ¶ÒÓœ" disk.nvme0c0n1.in‚ 711.06640625 disk.sda3.out‚ 278.1328125! disk.dm-0.out‚72503.865234375 disk.rbd3.in‚0( proc.memory.availableMB‚ 248.21484375 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd1.out‚ 3.7421875 disk.rbd0.out‚0 network.recv‚ 2.540314e+06 cpu‚25.36278636428993 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2879.48828125) disk./.usagePercent‚73.61310401598405$ memory_percent‚98.48501682281494 disk.sda.out‚ 278.1328125 disk.nvme0n1.in‚ 711.06640625 disk.dm-0.in‚ 711.06640625 disk.rbd0.in‚0 disk.rbd2.out‚ 162.53125 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚72503.865234375% disk./.usageGB‚1096.8884468078613& disk.nvme0c0n1.out‚72503.865234375 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 278.1328125 disk.rbd2.in‚0) proc.memory.percent‚17.57500171661377 network.sent‚ 2.2457451e+07}HŽ‘¼#‚(:† ö“äÔ“ù–™ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.978' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾ÿV&½#‚(:› ö“äÔ°«œ disk.drbd1777.out‚0& disk.nvme0n1.out‚72504.28271484375 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 711.06640625 disk.rbd3.out‚0% proc.memory.availableMB‚ 248.21875 network.sent‚ 2.2566439e+07 disk.sda1.out‚0# disk.dm-0.out‚72504.28271484375# proc.memory.rssMB‚ 2879.48828125 proc.cpu.threads‚17 disk.rbd0.out‚0( disk.nvme0c0n1.out‚72504.28271484375) disk./.usagePercent‚73.61310401598405 disk.sda1.in‚0 disk.dm-1.in‚0 cpu‚25.362128682199245% disk./.usageGB‚1096.8884468078613 disk.sda.out‚ 278.546875 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 3.7421875 disk.nvme0n1.in‚ 711.06640625 disk.dm-1.out‚ 278.546875" disk.nvme0c0n1.in‚ 711.06640625$ memory_percent‚98.48499298095703 network.recv‚ 2.547429e+06 disk.sda3.in‚0 disk.rbd2.in‚0 disk.sda2.out‚0) proc.memory.percent‚17.57500171661377 disk.rbd2.out‚ 162.62109375 disk.sda3.out‚ 278.546875 disk.sda.in‚0l’¦p2¾#- _wandbruntime‚7072  _runtime‚7072Ê oÈ¿# ' lr-AdamW/pg1‚1.3488356652498821e-05 ' lr-AdamW/pg2‚1.3488356652498821e-05 trainer/global_step‚9499 % _timestamp‚1.7884144620801384e+09 _runtime‚ 7072.49231748 _step‚777 ÜŸÏÍÀ#Ç trainer/global_step‚9499 % _timestamp‚1.7884144620801384e+09 _runtime‚ 7072.49231748 _step‚777 ' lr-AdamW/pg1‚1.3488356652498821e-05 ' lr-AdamW/pg2‚1.3488356652498821e-05‰x'ö¶2Á#- _wandbruntime‚7072  _runtime‚7072hñy³ë'Â#åO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 191881844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53752193 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72956204 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚267391175133184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13352490264922 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124993257 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _runtime‚ 7072.49326763 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚267391157130240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚254364677282816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚296952331784192 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13622736005120 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124993259 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124993259 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚280782248519168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚280782061516288 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126708935 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚778 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13391073385984 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13391073385984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 316895141 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13622938201088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚296952105648128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚283329393583104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚267717184587098 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202195968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚267717002426334 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124993257 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚9499 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53752729 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191881844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13352325143518 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788414462080922e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13390904386048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚280782061516288 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199±Àª Ã#êO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚296952105648128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚283329393583104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚267717184587098 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202195968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚267717002426334 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124993257 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚9499 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53752729 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191881844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13352325143518 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788414462080922e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13390904386048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚280782061516288 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 191881844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53752193 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72956204 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚267391175133184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13352490264922 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124993257 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _runtime‚ 7072.49326763 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚267391157130240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚254364677282816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚296952331784192 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13622736005120 60DeviceStatsMonitor.on_train_batch_st³kB`art/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124993259 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124993259 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚280782248519168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚280782061516288 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126708935 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚778 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13391073385984 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13391073385984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 316895141 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13622938201088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0Š4µ={2Ä#- _wandbruntime‚7073  _runtime‚7073ÍGïXö&Å#ðM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚280810911035392 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13353879429674 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚267418426494976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 191921695 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126721176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53757595 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚283358282591232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13392297524224 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚779 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72963583 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 191901631 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 trainer/global_step‚9499 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚267744516934186 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 316907547 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 316927613 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13392466537472 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125005916 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13624150519296 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 191901631 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 316907547 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72963581 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚267418444497920 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13392297524224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚280810724019200 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚779 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72963583 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 191901631 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 trainer/global_step‚9499 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚267744516934186 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 316907547 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 316927613 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13392466537472 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125005916 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13624150519296 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 191901631 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 316907547 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72963581 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚267418444497920 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13392297524224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚280810724019200 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126721176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53757595 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚283358282591232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13392297524224‹@¤12Ç#- _wandbruntime‚7073  _runtime‚7073ž¸ö[È#Õ 0loss/FRt4/assignment_loss‚3.4674328474894813 /loss/FRt4/detection_loss‚2.3176399735005004 1loss/SRqqt3/detection_loss‚1.3572509980294853 /loss/FRt1/detection_loss‚2.9731025880780715 /loss/SRqqt1/detection_loss‚2.58577429822513 /loss/FBt4/detection_loss‚2.6082411482930183 _runtime‚7073.154555861 _step‚780 /loss/FRt1/assignment_loss‚4.543440646138684 2loss/SRqqt2/assignment_loss‚1.2967234589790895 0loss/FBt1/assignment_loss‚0.3237028580414127 0loss/FRt2/assignment_loss‚2.9358005063100294 0loss/FBt2/assignment_loss‚0.5134565774351358 .loss/FBt2/detection_loss‚5.319987207651138 &loss/total_loss‚49.090740028449815 % _timestamp‚1.7884144627422204e+09 1loss/FBt4/assignment_loss‚0.32312920965269615 2loss/SRqqt3/assignment_loss‚0.9765447763493285 2loss/SRqqt4/assignment_loss‚1.7450226167285885 /loss/FRt3/detection_loss‚1.7782669947032006 /loss/FBt1/detection_loss‚2.6636895963123868 /loss/FRt3/assignment_loss‚2.421367965398296 0loss/SRqqt2/detection_loss‚1.528404087585918 1loss/SRqqt4/detection_loss‚1.5878055068077865 /loss/FRt2/detection_loss‚2.3250777193091134 2loss/SRqqt1/assignment_loss‚2.4784920843584195 0loss/FBt3/assignment_loss‚0.2228909323896306 epoch‚7 trainer/global_step‚9499 /loss/FBt3/detection_loss‚0.7974954306832719 AØ„`É#Ú 2loss/SRqqt1/assignment_loss‚2.4784920843584195 0loss/FBt3/assignment_loss‚0.2228909323896306 epoch‚7 trainer/global_step‚9499 /loss/FBt3/detection_loss‚0.7974954306832719 0loss/FRt4/assignment_loss‚3.4674328474894813 /loss/FRt4/detection_loss‚2.3176399735005004 1loss/SRqqt3/detection_loss‚1.3572509980294853 /loss/FRt1/detection_loss‚2.9731025880780715 /loss/SRqqt1/detection_loss‚2.58577429822513 /loss/FBt4/detection_loss‚2.6082411482930183 _runtime‚7073.154555861 _step‚780 /loss/FRt1/assignment_loss‚4.543440646138684 2loss/SRqqt2/assignment_loss‚1.2967234589790895 0loss/FBt1/assignment_loss‚0.3237028580414127 0loss/FRt2/assignment_loss‚2.9358005063100294 0loss/FBt2/assignment_loss‚0.5134565774351358 .loss/FBt2/detection_loss‚5.319987207651138 &loss/total_loss‚49.090740028449815 % _timestamp‚1.7884144627422204e+09 1loss/FBt4/assignment_loss‚0.32312920965269615 2loss/SRqqt3/assignment_loss‚0.9765447763493285 2loss/SRqqt4/assignment_loss‚1.7450226167285885 /loss/FRt3/detection_loss‚1.7782669947032006 /loss/FBt1/detection_loss‚2.6636895963123868 /loss/FRt3/assignment_loss‚2.421367965398296 0loss/SRqqt2/detection_loss‚1.528404087585918 1loss/SRqqt4/detection_loss‚1.5878055068077865 /loss/FRt2/detection_loss‚2.3250777193091134ŒO·‘Ê#‚(:† …”äÔ¶š»™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Îz\(Ë#‚(: …”äÔÍîž disk.drbd1777.in‚0 disk.rbd1.out‚ 3.7421875 disk.dm-1.in‚0 disk.dm-1.out‚ 279.2890625 disk.sda.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 711.06640625 disk.dm-0.in‚ 711.06640625 disk.rbd2.out‚ 162.62109375% proc.memory.availableMB‚ 247.96875# proc.memory.rssMB‚ 2879.48828125 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.rbd3.out‚0 network.sent‚ 2.2572613e+07$ disk./.usageGB‚1096.888442993164 disk.rbd0.in‚0$ memory_percent‚98.48651885986328 disk.sda1.out‚0 disk.drbd1777.out‚0& disk.nvme0n1.out‚72504.60302734375# disk.dm-0.out‚72504.60302734375 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda3.out‚ 279.2890625 disk.sda.out‚ 279.2890625 disk.rbd1.in‚0 disk.sda2.in‚0 network.recv‚ 2.552264e+06 disk.sda2.out‚0 disk.sda3.in‚0) proc.memory.percent‚17.57500171661377 cpu‚25.361496487118508" disk.nvme0c0n1.in‚ 711.06640625( disk.nvme0c0n1.out‚72504.60302734375) disk./.usagePercent‚73.61310375997651¯ò²ÿÌ#‚(:„ ””äÔ½¡²™ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.29' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.058 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Jó61Í#‚(:¦ ””äÔ†ßß›% disk./.usageGB‚1096.8964195251465 disk.drbd1777.out‚0! disk.rbd2.out‚162.72314453125" disk.nvme0c0n1.in‚ 711.06640625( proc.memory.availableMB‚ 247.51953125 disk.sda2.out‚0 disk.sda.out‚ 279.87890625( disk./.usagePercent‚73.6136390717306 disk.rbd1.out‚ 3.7421875 disk.dm-0.in‚ 711.06640625 disk.dm-1.out‚ 279.87890625 disk.sda1.in‚0 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2879.48828125 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 279.87890625$ memory_percent‚98.48926067352295 disk.drbd1777.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0 network.sent‚ 2.2680908e+07 disk.rbd2.in‚0( disk.nvme0c0n1.out‚72513.40380859375 disk.sda2.in‚0 disk.sda1.out‚0 network.recv‚ 2.558653e+06 disk.nvme0n1.in‚ 711.06640625& disk.nvme0n1.out‚72513.40380859375# disk.dm-0.out‚72513.40380859375) proc.memory.percent‚17.57500171661377 cpu‚25.360816107056525 proc.cpu.threads‚17e±×f2Î#- _wandbruntime‚7107  _runtime‚7107'¦ ÉÏ#à _step‚781 ' lr-AdamW/pg1‚1.2996154495428406e-05 ' lr-AdamW/pg2‚1.2996154495428406e-05 trainer/global_step‚9549 % _timestamp‚1.7884144975267997e+09 _runtime‚7107.938951616ªTCEÎÐ#È ' lr-AdamW/pg1‚1.2996154495428406e-05 ' lr-AdamW/pg2‚1.2996154495428406e-05 trainer/global_step‚9549 % _timestamp‚1.7884144975267997e+09 _runtime‚7107.938951616 _step‚781ýÁåE2Ñ#- _wandbruntime‚7107  _runtime‚7107UÚ„Z¬Ò#æO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127349584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13693635385344 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13421962083330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚7107.939883161 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚268754643369984 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 192894584 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125626207 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54025072 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13460575708672 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚9549 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚255661836399616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚268754625367040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 192874546 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _step‚782 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125626207 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚298468652439552 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125626209 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚282215201075712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 318520793 XKDeviceStatsMonitor.on_train_batch_start/inactive_sò³Â@plit.large_pool.allocated‚73325056 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13460744708608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 192874546 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 318500753 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127350128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54024530 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚298468884867072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚269083815522306 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚268754643369984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚255661853438976 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13460575708672 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚255661836399616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚268754625367040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 192874546 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _step‚782 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125626207 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚298468652439552 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125626209 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚282215201075712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 318520793 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73325056 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13460744708608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 192874546 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 318500753 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127350128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54024530 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚298468884867072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚269083815522306 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚268754643369984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚255661853438976 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13460575708672 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚7107.939883161 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚268754643369984 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006ŽÑ5nv2Ô#- _wandbruntime‚7108  _runtime‚7108Ff†.÷&Õ#ñM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚255687779581952 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _step‚783 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192914397 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚268781912734720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318553265 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13423351248082 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13462137860096 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 192894333 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚284803953942528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13461968846848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚282243863578624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚268781912734720 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125638868 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 192894333 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13695045260800 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13462137860096 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127362395 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13461968846848 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54029964 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125638866 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚284803930002432 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚255687796621312 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13462137860096 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127362395 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13461968846848 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54029964 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125638866 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚284803930002432 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚255687796621312 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚255687779581952 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _step‚783 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192914397 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚268781912734720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318553265 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13423351248082 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13462137860096 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 192894333 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚284803953942528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13461968846848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚282243863578624 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚268781912734720 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125638868 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992¥B÷<2×#- _wandbruntime‚7108  _runtime‚7108… zZØ#Ô 1loss/FBt2/assignment_loss‚0.39092955253242206 .loss/FBt2/detection_loss‚4.187031984329224 /loss/FRt1/assignment_loss‚4.943982733620538 trainer/global_step‚9549 1loss/FBt3/assignment_loss‚0.21338636749343012 /loss/FRt2/assignment_loss‚3.243817804948143 0loss/FBt1/assignment_loss‚0.5027131200804433 /loss/FRt3/detection_loss‚1.5942643041706692 1loss/SRqqt2/detection_loss‚1.5059781494048925 /loss/FBt1/detection_loss‚2.4632401218016944 0loss/FBt4/assignment_loss‚0.4960355975034482 .loss/FRt1/detection_loss‚3.011390180499465 epoch‚7 _runtime‚7108.593053413 _step‚784 .loss/FBt4/detection_loss‚2.237560444644519 /loss/FRt2/detection_loss‚2.3148235037274985 0loss/FRt3/assignment_loss‚1.7738454604552965 /loss/FRt4/assignment_loss‚3.336466344627174 2loss/SRqqt4/assignment_loss‚1.7827596905856178 1loss/SRqqt1/detection_loss‚2.3345987376044777 2loss/SRqqt1/assignment_loss‚2.4259857456473743 .loss/FRt4/detection_loss‚2.695666202016779 %loss/total_loss‚48.15145511125061 2loss/SRqqt3/assignment_loss‚1.3986758192614563 1loss/SRqqt4/detection_loss‚1.8517936975336993 % _timestamp‚1.7884144981810038e+09 1loss/SRqqt3/detection_loss‚1.1376085015681554 /loss/FBt3/detection_loss‚1.0127437084374198 2loss/SRqqt2/assignment_loss‚1.2961573387567813™NM¿_Ù#Ù /loss/FRt1/assignment_loss‚4.943982733620538 trainer/global_step‚9549 1loss/FBt3/assignment_loss‚0.21338636749343012 /loss/FRt2/assignment_loss‚3.243817804948143 0loss/FBt1/assignment_loss‚0.5027131200804433 /loss/FRt3/detection_loss‚1.5942643041706692 1loss/SRqqt2/detection_loss‚1.5059781494048925 /loss/FBt1/detection_loss‚2.4632401218016944 0loss/FBt4/assignment_loss‚0.4960355975034482 .loss/FRt1/detection_loss‚3.011390180499465 epoch‚7 _runtime‚7108.593053413 _step‚784 .loss/FBt4/detection_loss‚2.237560444644519 /loss/FRt2/detection_loss‚2.3148235037274985 0loss/FRt3/assignment_loss‚1.7738454604552965 /loss/FRt4/assignment_loss‚3.336466344627174 2loss/SRqqt4/assignment_loss‚1.7827596905856178 1loss/SRqqt1/detection_loss‚2.3345987376044777 2loss/SRqqt1/assignment_loss‚2.4259857456473743 .loss/FRt4/detection_loss‚2.695666202016779 %loss/total_loss‚48.15145511125061 2loss/SRqqt3/assignment_loss‚1.3986758192614563 1loss/SRqqt4/detection_loss‚1.8517936975336993 % _timestamp‚1.7884144981810038e+09 1loss/SRqqt3/detection_loss‚1.1376085015681554 /loss/FBt3/detection_loss‚1.0127437084374198 2loss/SRqqt2/assignment_loss‚1.2961573387567813 1loss/FBt2/assignment_loss‚0.39092955253242206 .loss/FBt2/detection_loss‚4.187031984329224­ÞC›Ú#‚(: £”äÔÉê¡™ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚89.096' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.819200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿þ¤ Û#‚(:• £”äÔ½à› disk.sda.in‚0 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 711.06640625 disk.rbd0.out‚0 network.recv‚ 2.562478e+06' disk.nvme0c0n1.out‚72513.8466796875 disk.sda2.out‚0 disk.sda.out‚280.4375 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 711.06640625 disk.sda3.out‚280.4375 disk.dm-1.out‚280.4375 disk.rbd3.in‚0$ memory_percent‚98.48947525024414 proc.cpu.threads‚17 disk.sda2.in‚0" disk.dm-0.out‚72513.8466796875# proc.memory.rssMB‚ 2879.48828125$ disk./.usageGB‚1096.897361755371 disk.sda3.in‚0% disk.nvme0n1.out‚72513.8466796875 network.sent‚ 2.2686305e+07 disk.drbd1777.in‚0 disk.rbd1.out‚ 3.7421875) proc.memory.percent‚17.57500171661377 cpu‚25.360127557295343) disk./.usagePercent‚73.61370230559157 disk.rbd0.in‚0 disk.rbd3.out‚0& proc.memory.availableMB‚ 247.484375 disk.nvme0n1.in‚ 711.06640625! disk.rbd2.out‚162.72314453125âyrœÜ#‚(:‘ ²”äÔ¢€¢™ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.962' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.592399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M:ýÉ*Ý#‚(:Ÿ ²”äÔçæÊœ$ disk.nvme0n1.out‚72514.755859375 network.sent‚ 2.2795435e+07 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 3.7421875 disk.dm-1.in‚0) proc.memory.percent‚17.57500171661377( disk./.usagePercent‚73.6137010255539! disk.rbd2.out‚164.37548828125 disk.sda.in‚0# memory_percent‚98.4879732131958" disk.nvme0c0n1.in‚ 711.06640625 disk.rbd0.in‚0( proc.memory.availableMB‚ 247.73046875! disk.dm-0.out‚72514.755859375 disk.rbd2.in‚0& disk.nvme0c0n1.out‚72514.755859375% disk./.usageGB‚1096.8973426818848 disk.sda3.out‚ 280.82421875 network.recv‚ 2.569684e+06 disk.nvme0n1.in‚ 711.06640625 disk.dm-0.in‚ 711.06640625 disk.dm-1.out‚ 280.82421875# proc.memory.rssMB‚ 2879.48828125 cpu‚25.359505806959827 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda.out‚ 280.82421875 disk.sda1.in‚0`Ú‘Þ#‚(:† Á”äÔåíÀ™ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.822' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽO/ß#‚(:¤ Á”äÔý¡Úœ disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 711.06640625 proc.cpu.threads‚17 network.sent‚ 2.2801557e+07# disk.dm-0.out‚72613.94189453125 disk.dm-1.out‚ 281.35546875 disk.drbd1777.in‚0& disk.nvme0n1.out‚72613.94189453125 disk.rbd1.out‚ 3.7421875 disk.dm-0.in‚ 711.06640625 disk.dm-1.in‚0( disk.nvme0c0n1.out‚72613.94189453125% disk./.usageGB‚1097.0222816467285 disk.rbd0.in‚0 disk.sda3.out‚ 281.35546875) disk./.usagePercent‚73.62208578431533 disk.sda.out‚ 281.35546875 disk.rbd1.in‚0 network.recv‚ 2.573858e+06 disk.rbd0.out‚0! disk.rbd2.out‚164.37548828125 disk.sda.in‚0 disk.sda1.out‚0% proc.memory.availableMB‚ 247.46875# proc.memory.rssMB‚ 2879.48828125) proc.memory.percent‚17.57500171661377 disk.rbd2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 711.06640625$ memory_percent‚98.48957061767578 cpu‚25.358856910797194 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda3.in‚0×b‡2à#- _wandbruntime‚7143  _runtime‚7143§éú‡Éá#à ' lr-AdamW/pg2‚1.2511771707089774e-05 trainer/global_step‚9599 % _timestamp‚1.7884145325970647e+09 _runtime‚7143.009252907 _step‚785 ' lr-AdamW/pg1‚1.2511771707089774e-05sÔqáÎâ#È _runtime‚7143.009252907 _step‚785 ' lr-AdamW/pg1‚1.2511771707089774e-05 ' lr-AdamW/pg2‚1.2511771707089774e-05 trainer/global_step‚9599 % _timestamp‚1.7884145325970647e+09‘£€Í2ã#- _wandbruntime‚7143  _runtime‚71432-¿åì'ä#æO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 193867248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚283648340635136 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚7143.010285883 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73693904 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚256958995516416 _step‚786 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚546 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13530416031232 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13530247031296 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚283648340635136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73693906 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚286220688405504 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 $ _timestamp‚1.788414532597851e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚270118111606784 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚299985196455424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚9599 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚283648527638016 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13764740477440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13491268780334 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚299985428882944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚270118093603840 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 193887286 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54297093 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 193887286 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54296549 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚270118111606784 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126259157 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127990999 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚286220664465408 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚270450446457514 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚270450264296750 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13530247031296 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚256959012555776 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13491433901738 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 193867248/d´Ññ'å#ëO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚270118111606784 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126259157 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127990999 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚286220664465408 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚270450446457514 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚270450264296750 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13530247031296 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚256959012555776 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13491433901738 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 193867248 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 193867248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚283648340635136 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚7143.010285883 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73693904 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚256958995516416 _step‚786 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚546 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13530416031232 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13530247031296 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚283648340635136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73693906 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚286220688405504 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 $ _timestamp‚1.788414532597851e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚270118111606784 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚299985196455424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚9599 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚283648527638016 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13764740477440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13491268780334 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚299985428882944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚270118093603840 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 193887286 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54297093 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 193887286 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54296549 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416’?Œ+2æ#- _wandbruntime‚7143  _runtime‚71433ÇçÈÓç#ñM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚283677003138048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13531809182720 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13531640169472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73701281 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13765943652352 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚256984938698752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13531809182720 E=DeviceStatsMonj3Wg$"itor.on_train_batch_end/segment.small_pool.peak‚199 trainer/global_step‚9599 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13492823066490 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73701283 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚286249577413632 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 320178917 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13531640169472 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚270477778804602 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126271818 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 193907099 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256984955738112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 % _timestamp‚1.7884145332725213e+09 _runtime‚7143.685257095 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚283677190154240 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128003063 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 193887035 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚283677190154240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54302317 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 320158851 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚270145380971520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚270145362968576 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126271816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54301782 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13492657944878 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 128003600 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚286249601353728 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚787 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 320158851 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126271818 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320178917 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 193887035 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚300015521065984 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128003063 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 193887035 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚283677190154240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54302317 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 320158851 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚270145380971520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚270145362968576 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126271816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54301782 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13492657944878 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 128003600 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚286249601353728 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚787 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 320158851 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126271818 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320178917 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 193887035 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚300015521065984 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13531809182720 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13531640169472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73701281 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13765943652352 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚256984938698752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13531809182720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 trainer/global_step‚9599 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13492823066490 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73701283 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚286249577413632 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 320178917 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13531640169472 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚270477778804602 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126271818 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 193907099 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256984955738112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 % _timestamp‚1.7884145332725213e+09 _runtime‚7143.685257095 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚283677190154240“Ú ¦©2é#- _wandbruntime‚7143  _runtime‚7143קP5Wê#Ñ 1loss/SRqqt4/detection_loss‚2.0092981963728866 0loss/FBt2/assignment_loss‚0.3673985470086336 /loss/FRt2/detection_loss‚2.2841322449967265 /loss/FBt3/detection_loss‚0.5866279975424338 trainer/global_step‚9599 2loss/SRqqt3/assignment_loss‚1.1745986477563195 0loss/FBt3/assignment_loss‚0.2776194698528826 /loss/FBt4/assignment_loss‚0.591345947685373 % _timestamp‚1.7884145332744062e+09 0loss/FRt4/assignment_loss‚3.6255478580792744 /loss/FRt3/detection_loss‚2.2978300502442797 /loss/FRt4/detection_loss‚1.9319387177626317 /loss/FBt4/detection_loss‚2.0750105593885695 %loss/total_loss‚47.78698046660142 /loss/FRt1/assignment_loss‚4.345867409850612 0loss/FBt1/assignment_loss‚0.3999582517551192 0loss/SRqqt1/detection_loss‚2.858125078678131 0loss/SRqqt3/detection_loss‚1.449981362211938 /loss/FBt1/detection_loss‚2.7232611817973003 2loss/SRqqt1/assignment_loss‚2.7522587251663206 .loss/FRt1/detection_loss‚2.912812708002148 epoch‚7 _runtime‚7143.686451014 0loss/FRt3/assignment_loss‚2.2541398017302803 2loss/SRqqt4/assignment_loss‚1.7458745947418113 1loss/SRqqt2/detection_loss‚1.7221360639447258 .loss/FBt2/detection_loss‚2.802514056364695 _step‚788 /loss/FRt2/assignment_loss‚3.289105753103892 1loss/SRqqt2/assignment_loss‚1.309597242564436i€ëÄ\ë#Ö /loss/FRt1/assignment_loss‚4.345867409850612 0loss/FBt1/assignment_loss‚0.3999582517551192 0loss/SRqqt1/detection_loss‚2.858125078678131 0loss/SRqqt3/detection_loss‚1.449981362211938 /loss/FBt1/detection_loss‚2.7232611817973003 2loss/SRqqt1/assignment_loss‚2.7522587251663206 .loss/FRt1/detection_loss‚2.912812708002148 epoch‚7 _runtime‚7143.686451014 0loss/FRt3/assignment_loss‚2.2541398017302803 2loss/SRqqt4/assignment_loss‚1.7458745947418113 1loss/SRqqt2/detection_loss‚1.7221360639447258 .loss/FBt2/detection_loss‚2.802514056364695 _step‚788 /loss/FRt2/assignment_loss‚3.289105753103892 1loss/SRqqt2/assignment_loss‚1.309597242564436 1loss/SRqqt4/detection_loss‚2.0092981963728866 0loss/FBt2/assignment_loss‚0.3673985470086336 /loss/FRt2/detection_loss‚2.2841322449967265 /loss/FBt3/detection_loss‚0.5866279975424338 trainer/global_step‚9599 2loss/SRqqt3/assignment_loss‚1.1745986477563195 0loss/FBt3/assignment_loss‚0.2776194698528826 /loss/FBt4/assignment_loss‚0.591345947685373 % _timestamp‚1.7884145332744062e+09 0loss/FRt4/assignment_loss‚3.6255478580792744 /loss/FRt3/detection_loss‚2.2978300502442797 /loss/FRt4/detection_loss‚1.9319387177626317 /loss/FBt4/detection_loss‚2.0750105593885695 %loss/total_loss‚47.78698046660142” Ûð‘ì#‚(:† ДäÔù¬¯™ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.322' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øìá$í#‚(:™ ДäÔ²•úœ) disk./.usagePercent‚73.63051867249321" disk.nvme0c0n1.in‚ 711.06640625% disk./.usageGB‚1097.1479377746582 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda3.in‚0$ memory_percent‚98.48825931549072 disk.sda.out‚ 281.99609375 disk.sda1.in‚0 network.recv‚ 2.578042e+06 disk.dm-0.in‚ 711.06640625 disk.dm-0.out‚ 72614.5078125 disk.rbd3.out‚0 disk.sda3.out‚ 281.99609375 disk.drbd1777.in‚0 disk.rbd1.out‚ 3.7421875 disk.dm-1.out‚ 281.99609375 disk.rbd2.out‚164.4775390625$ disk.nvme0c0n1.out‚ 72614.5078125 disk.sda2.out‚0# proc.memory.rssMB‚ 2879.48828125 proc.cpu.threads‚17 disk.rbd0.in‚0 network.sent‚ 2.2806894e+07 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.drbd1777.out‚0( proc.memory.availableMB‚ 247.68359375 disk.nvme0n1.in‚ 711.06640625 disk.dm-1.in‚0 disk.rbd2.in‚0) proc.memory.percent‚17.57500171661377" disk.nvme0n1.out‚ 72614.5078125 disk.sda1.out‚0 cpu‚25.35824464386297]ºKÏ‘î#‚(:† ß”äÔÅ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Úhñ,ï#‚(:¡ ß”äÔ½øÜ› disk.drbd1777.in‚0 disk.sda3.out‚ 282.62109375 disk.nvme0n1.in‚ 711.06640625 network.sent‚ 2.2915135e+07 disk.rbd3.out‚0 disk.sda.in‚0 disk.drbd1777.out‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda1.out‚0) disk./.usagePercent‚73.63048820759663% disk./.usageGB‚1097.1474838256836 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 711.06640625 disk.rbd1.out‚ 3.7421875 disk.dm-0.in‚ 711.06640625 network.recv‚ 2.584155e+06 disk.rbd2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0" disk.dm-0.out‚72713.6923828125 disk.dm-1.out‚ 282.62109375$ memory_percent‚98.48833084106445& proc.memory.availableMB‚ 247.671875' disk.nvme0c0n1.out‚72713.6923828125% disk.nvme0n1.out‚72713.6923828125# proc.memory.rssMB‚ 2879.48828125) proc.memory.percent‚17.57500171661377 cpu‚25.357576162445568 disk.rbd0.in‚0 disk.rbd2.out‚164.4775390625 disk.sda3.in‚0 disk.sda.out‚ 282.62109375 disk.sda1.in‚0 proc.cpu.threads‚17z-5ê2ð#- _wandbruntime‚7178  _runtime‚7178•tÉñ#à ' lr-AdamW/pg2‚1.2035309232130024e-05 trainer/global_step‚9649 % _timestamp‚1.7884145682186875e+09 _runtime‚7178.630842533 _step‚789 ' lr-AdamW/pg1‚1.2035309232130024e-05=}ÃÎò#È trainer/global_step‚9649 % _timestamp‚1.7884145682186875e+09 _runtime‚7178.630842533 _step‚789 ' lr-AdamW/pg1‚1.2035309232130024e-05 ' lr-AdamW/pg2‚1.2035309232130024e-05•Z¬ 2ó#- _wandbruntime‚7178  _runtime‚7178·åO,ô#åO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 194859950 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13599918353920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚301501967855616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13600087353856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13560905720146 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚285081480194560 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13835423551488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _runtime‚ 7178.63190763 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126892107 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 194879988 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 194879988 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚271816895231958 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚271481561840640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚285081667197440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 321772097 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54568555 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 trainer/global_step‚9649 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 128631311 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 194859950 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚258256154633216 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126892107 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚258256171672576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 321752057 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚271481579843584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74062756 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚287666335816704 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13835632038912 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚271481561840640 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _step‚790 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13599918353920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 R>DeviceStatsMonitor.on_train_batch_start/active_by¦¯‘¿ tes.all.freed‚285081480194560 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13600087353856 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚546 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚301501735428096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚287666311876608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2* .ð'õ#êO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126892107 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 194879988 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 194879988 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚271816895231958 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚271481561840640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚285081667197440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 321772097 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54568555 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 trainer/global_step‚9649 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 128631311 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 194859950 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚258256154633216 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126892107 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚258256171672576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 321752057 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚271481579843584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74062756 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚287666335816704 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13835632038912 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚271481561840640 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _step‚790 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13599918353920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚285081480194560 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13600087353856 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚546 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚301501735428096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚287666311876608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 194859950 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13599918353920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚301501967855616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13600087353856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13560905720146 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚285081480194560 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13835423551488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _runtime‚ 7178.63190763 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0–BÈgm2ö#- _wandbruntime‚7179  _runtime‚7179,öö&÷#ðM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74070133 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 321784503 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126904768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 194879737 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 194879737 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚285110329713664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126904766 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚ 7179.30286971 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13837081028096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚287695248764928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 194899801 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚9649 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126904768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13562129763286 % _timestamp‚1.7884145688900497e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 321804569 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 321784503 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚287695224824832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13562294884898 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74070131 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚271508849208320 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚271508831205376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13601480505344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13836874651136 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚271844409739810 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13601480505344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 321804569 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 194899801 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚791 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚271508831205376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54573927 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126904766 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 128644060 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54573403 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚285110142697472 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚301532099475968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚258282114854912 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚301532329793024 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚285110329713664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128643534 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74070131 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚271508849208320 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚271508831205376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13601480505344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13836874651136 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚271844409739810 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13601480505344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 321804569 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 194899801 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚791 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚271508831205376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54573927 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126904766 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 128644060 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54573403 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚285110142697472 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚301532099475968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚258282114854912 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚301532329793024 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚285110329713664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128643534 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 321784503 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126904768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 194879737 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 194879737 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚285110329713664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126904766 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚ 7179.30286971 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13837081028096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚287695248764928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 194899801 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚9649 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126904768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13562129763286 % _timestamp‚1.7884145688900497e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 321804569 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 321784503 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚287695224824832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13562294884898 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 127708154 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 4541568000 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 269877760 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚289291884398592 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 323405785 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 4757245257 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 4390226944 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚13609908759198 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚286682424881152 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚26 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚273339949755038 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚416 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚217 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚54938564 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 4912020992 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1006 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚303177339188224 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 145311232 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚416 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884146321704166e+09 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚74550927 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚273033169336320 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2460 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 406789969 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 5041553408 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚13649128236544 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚54938282 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚259730040995840 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 195697631 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 127708154 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 4541568000 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 269877760 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚289291884398592 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 323405785 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 4757245257 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 4390226944 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚13609908759198 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚286682424881152 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚26 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚273339949755038 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚416 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚217 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚54938564 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 4912020992 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1006 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚303177339188224 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 145311232 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚416 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884146321704166e+09 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚74550927 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚273033169336320 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2460 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 406789969 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 5041553408 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚13649128236544 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚54938282 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚259730040995840 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 195697631 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 4912020992 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚217 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚199 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 127718864 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2460 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚303197291889664 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 5041553408 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚985 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚303197156182528 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚13649604680704 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚74557427 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚416 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚199 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚217 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 140964952 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 406789969 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚286701639202816 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚273052034522112 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚273052016519168 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 127308288 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚26 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚289311511904256 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚286701493891584 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 145311232 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚285 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1006 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚273358057950086 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 4757245257 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 4541568000 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 123925592 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚13649477372416 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚259747942549504 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚13610132439942 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 417333248 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 129500686 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 127718864 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚259747925510144 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 4912020992 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚217 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚199 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 127718864 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2460 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚303197291889664 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 5041553408 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚985 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚303197156182528 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚13649604680704 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚74557427 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚416 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚199 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚217 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 140964952 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 406789969 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚286701639202816 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚273052034522112 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚273052016519168 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 127308288 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚26 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0šìŽ.ý›Š$‚(: ª•äÔ°“ò™ gpu.0.gpu‚39.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.563' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.712600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B#›©0‹$‚(:¥ ª•äÔ§ÀÞ cpu‚25.391632342415964 disk.dm-1.out‚ 285.61328125 disk.rbd2.in‚0$ memory_percent‚98.49193096160889 network.sent‚ 2.3050603e+07 disk.rbd3.out‚0 disk.sda2.out‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 247.08203125) proc.memory.percent‚17.57500171661377 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda.in‚0" disk.nvme0c0n1.in‚ 711.06640625 disk.sda2.in‚0 disk.nvme0n1.in‚ 711.06640625# disk.dm-0.out‚73115.31298828125! disk.rbd2.out‚166.35693359375# disk./.usageGB‚1097.53564453125 disk.sda3.in‚0 disk.sda.out‚ 285.61328125 disk.rbd0.in‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚73115.31298828125 disk.drbd1777.in‚0 disk.dm-0.in‚ 711.06640625# proc.memory.rssMB‚ 2879.48828125 network.recv‚ 2.614683e+06 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚73115.31298828125 disk.rbd0.out‚0 disk.sda3.out‚ 285.61328125) disk./.usagePercent‚73.65653799823572 disk.rbd1.out‚ 3.7421875øcÅó2Œ$- _wandbruntime‚7255  _runtime‚7255Å…óÏ/$û§ 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5428982377052307 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.32187503576278687 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04434523731470108 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9908046126365662 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19702506065368652 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.11302681267261505 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5341721773147583 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5333333611488342 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365078806877136 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13253799080848694 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2315654307603836 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5039799809455872 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.05284504592418671 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_4_of_7‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2993308901786804 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.39176368713378906 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6454666256904602 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13862499594688416 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09259258955717087 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5104166865348816 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14019601047039032 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.180568665266037 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.014833711087703705 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.34907403588294983 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03407225012779236 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9887880682945251 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.997589111328125 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.46997126936912537 particle/accuracy_3_of_5‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333283662796 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_3‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333134651184 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.03333333507180214 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.23362067341804504 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333283662796 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3725399971008301 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9806174039840698 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14019601047039032 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6818181872367859 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,particle/sensitivity‚0.30944302678108215 jet/accuracy_6_of_7‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4259258806705475 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14372164011001587 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.36952754855155945 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5272532105445862 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5186959505081177 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4734653830528259 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.032894738018512726 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2638888955116272 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.988676130771637 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.35943567752838135 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4127902686595917 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6818181872367859 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_6_of_6‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12536823749542236 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5077348351478577 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06116614490747452 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2503760755062103 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_4_of_5‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.37437403202056885 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5196220874786377 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5101422667503357 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1470373272895813 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.17342343926429749 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 +jet/accuracy_2_of_2‚0.11163841187953949 particle/accuracy_1_of_5‚1 *jet/accuracy_1_of_6‚0.9615384340286255 jet/accuracy_2_of_7‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.47908252477645874 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 *jet/accuracy_1_of_4‚0.8902087211608887 +jet/accuracy_2_of_5‚0.47604966163635254 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_6‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.39300066232681274 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17898313701152802 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9789473414421082 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06528396904468536 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4622930884361267 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.30000001192092896 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5221213698387146 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19403451681137085 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.48892858624458313 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1818181872367859 *jet/accuracy_1_of_3‚0.8535010814666748 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 jet/accuracy_3_of_7‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9859656691551208 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.19290123879909515 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1088888868689537 jet/accuracy_5_of_6‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5273133516311646 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5087466835975647 *jet/accuracy_3_of_5‚0.1254352182149887 particle/accuracy_2_of_3‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_6‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5057052969932556 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_puriœ|ia3dty‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 epoch‚7 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5069444179534912 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 % _timestamp‚1.7884146454811726e+09 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20410846173763275 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1269841343164444 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4371873736381531 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4871377646923065 7"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.48846435546875 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.36294645071029663 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_6_of_6‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.27734455466270447 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1186162605881691 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4480404257774353 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9667499661445618 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02928125485777855 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10422195494174957 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 trainer/global_step‚9671 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9849937558174133 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.023582765832543373 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.980228066444397 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5443704724311829 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.49001389741897583 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9906657338142395 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4541647732257843 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5127032399177551 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13152240216732025 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9795649647712708 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9890435934066772 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 ,jet/accuracy_3_of_3‚0.020156657323241234 particle/accuracy_4_of_4‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_6‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14207322895526886 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.38377195596694946 jet/accuracy_5_of_5‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4791666567325592 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8270692229270935 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1101568415760994 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4715725779533386 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6111111640930176 particle/accuracy_5_of_6‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6111111044883728 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.00947341974824667 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13620245456695557 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1666666567325592 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19028188288211823 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5178571343421936 _step‚795 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19901007413864136 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.551793098449707 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16664861142635345 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.40220633149147034 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14431947469711304 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.39760351181030273 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5346882343292236 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5715259313583374 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0054644811898469925 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0032051282469183207 (particle/accuracy‚0.8499245643615723 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2675926089286804 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.46384158730506897 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.30706536769866943 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.519444465637207 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5413690209388733 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18799112737178802 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17574551701545715 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5369536280632019 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6808272004127502 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4262986481189728 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3726402223110199 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12952980399131775 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9934807419776917 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5373362302780151 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.32910171151161194 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.257953017950058 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.013844085857272148 particle/accuracy_3_of_3‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1268235445022583 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.45715078711509705 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 particle/accuracy_7_of_7‚1 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14632156491279602 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_6‚0.09615384787321091 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 *jet/accuracy_1_of_5‚0.9345813989639282 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9859079718589783 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5384479761123657 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12566542625427246 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.34090909361839294 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9920848608016968 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4470527470111847 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4722132682800293 )jet/accuracy_1_of_2‚0.738539457321167 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9930710196495056 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.19122806191444397 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5541219115257263 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9960625171661377 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5588235259056091 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2158748060464859 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5092592835426331 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.22398674488067627 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3465277850627899 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.558548629283905 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0010582010727375746 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_4_of_7‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06535883247852325 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.19770115613937378 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.30366894602775574 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0037740038242191076 -jet/accuracy_4_of_5‚0.0026881720405071974 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40861591696739197 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3782828450202942 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16583894193172455 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14799366891384125 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.025956125929951668 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3261514902114868 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0035714285913854837 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5057471394538879 particle/accuracy_1_of_1‚1 (particle/f_score‚0.39147573709487915 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184397459030151 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.09375 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10283688455820084 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4731394052505493 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.015100721269845963 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9433333873748779 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2337731420993805 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 particle/accuracy_3_of_4‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 _runtime‚7255.895654012 6 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.162052720785141 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009635006077587605 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 +particle/specificity‚0.9499168395996094 jet/accuracy_1_of_7‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23084814846515656 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24242034554481506 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06955762952566147 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3653256595134735 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 particle/accuracy_6_of_7‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_2_of_4‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7"Purity/0FRt1SRqqt0FBt/event_purity‚0.48846435546875 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.509023129940033 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9855158925056458 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3432180881500244 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 *jet/accuracy_2_of_6‚0.5320513248443604 particle/accuracy_5_of_7‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3798888921737671 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04240810498595238 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16593800485134125 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2569444477558136 particle/accuracy_3_of_7‚1 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.023149866610765457 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 particle/accuracy_1_of_7‚1 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11348631232976913 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9829162359237671 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08398276567459106 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5625 6 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.356589138507843 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06578947603702545 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 particle/accuracy_3_of_6‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6052631735801697 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993895292282104 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9873238205909729 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 +jet/accuracy_4_of_6‚0.01923076994717121 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9879252314567566 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5429945588111877 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.38768768310546875 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4512297809123993 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9628472924232483 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 jet/accuracy_5_of_7‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5636776685714722 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9655171632766724 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9704850316047668 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5526315569877625 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚47.69878651575614 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17393727600574493 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.39803916215896606 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9942327737808228 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.44558045268058777 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.184523805975914 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4791666567325592 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02077004499733448 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.029069768264889717 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5179159641265869 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5035853385925293 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4687500596046448 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.32997167110443115 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3516569435596466 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03679245337843895 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 +jet/accuracy_2_of_3‚0.23784992098808289 particle/accuracy_1_of_2‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_2_of_5‚1‹TR¿Ž$€¨ 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2503760755062103 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_4_of_5‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.37437403202056885 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5196220874786377 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5101422667503357 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1470373272895813 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.17342343926429749 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 +jet/accuracy_2_of_2‚0.11163841187953949 particle/accuracy_1_of_5‚1 *jet/accuracy_1_of_6‚0.9615384340286255 jet/accuracy_2_of_7‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.47908252477645874 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 *jet/accuracy_1_of_4‚0.8902087211608887 +jet/accuracy_2_of_5‚0.47604966163635254 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_6‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.39300066232681274 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17898313701152802 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9789473414421082 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06528396904468536 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4622930884361267 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.30000001192092896 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5221213698387146 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19403451681137085 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.48892858624458313 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1818181872367859 *jet/accuracy_1_of_3‚0.8535010814666748 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 jet/accuracy_3_of_7‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9859656691551208 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.19290123879909515 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1088888868689537 jet/accuracy_5_of_6‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5273133516311646 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5087466835975647 *jet/accuracy_3_of_5‚0.1254352182149887 particle/accuracy_2_of_3‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_6‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5057052969932556 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 epoch‚7 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5069444179534912 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 % _timestamp‚1.7884146454811726e+09 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20410846173763275 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1269841343164444 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4371873736381531 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4871377646923065 7"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.48846435546875 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.36294645071029663 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_6_of_6‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.27734455466270447 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1186162605881691 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4480404257774353 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9667499661445618 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02928125485777855 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10422195494174957 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 trainer/global_step‚9671 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9849937558174133 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.023582765832543373 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.980228066444397 9"Purity/0FRt*SRqqt2FBt/SRqRÖÿ¥Hxqt_purity‚0.5443704724311829 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.49001389741897583 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9906657338142395 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4541647732257843 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5127032399177551 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13152240216732025 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9795649647712708 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9890435934066772 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 ,jet/accuracy_3_of_3‚0.020156657323241234 particle/accuracy_4_of_4‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_6‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14207322895526886 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.38377195596694946 jet/accuracy_5_of_5‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4791666567325592 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8270692229270935 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1101568415760994 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4715725779533386 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6111111640930176 particle/accuracy_5_of_6‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6111111044883728 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.00947341974824667 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13620245456695557 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1666666567325592 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19028188288211823 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5178571343421936 _step‚795 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19901007413864136 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.551793098449707 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16664861142635345 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.40220633149147034 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14431947469711304 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.39760351181030273 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5346882343292236 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5715259313583374 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0054644811898469925 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0032051282469183207 (particle/accuracy‚0.8499245643615723 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2675926089286804 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.46384158730506897 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.30706536769866943 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.519444465637207 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5413690209388733 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18799112737178802 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17574551701545715 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5369536280632019 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6808272004127502 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4262986481189728 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3726402223110199 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12952980399131775 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9934807419776917 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5373362302780151 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.32910171151161194 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.257953017950058 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.013844085857272148 particle/accuracy_3_of_3‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1268235445022583 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.45715078711509705 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 particle/accuracy_7_of_7‚1 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14632156491279602 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_6‚0.09615384787321091 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 *jet/accuracy_1_of_5‚0.9345813989639282 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9859079718589783 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5384479761123657 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12566542625427246 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.34090909361839294 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9920848608016968 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4470527470111847 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4722132682800293 )jet/accuracy_1_of_2‚0.738539457321167 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9930710196495056 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.19122806191444397 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5541219115257263 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9960625171661377 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5588235259056091 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2158748060464859 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5092592835426331 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.22398674488067627 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3465277850627899 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.558548629283905 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0010582010727375746 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_4_of_7‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06535883247852325 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.19770115613937378 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.30366894602775574 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0037740038242191076 -jet/accuracy_4_of_5‚0.0026881720405071974 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40861591696739197 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3782828450202942 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16583894193172455 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14799366891384125 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.025956125929951668 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3261514902114868 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0035714285913854837 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5057471394538879 particle/accuracy_1_of_1‚1 (particle/f_score‚0.39147573709487915 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184397459030151 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.09375 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10283688455820084 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4731394052505493 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.015100721269845963 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9433333873748779 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2337731420993805 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 particle/accuracy_3_of_4‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 _runtime‚7255.895654012 6 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.162052720785141 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009635006077587605 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 +particle/specificity‚0.9499168395996094 jet/accuracy_1_of_7‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23084814846515656 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24242034554481506 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06955762952566147 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3653256595134735 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 particle/accuracy_6_of_7‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_2_of_4‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7"Purity/0FRt1SRqqt0FBt/event_purity‚0.48846435546875 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.509023129940033 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9855158925056458 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3432180881500244 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 *jet/accuracy_2_of_6‚0.5320513248443604 particle/accuracy_5_of_7‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3798888921737671 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04240810498595238 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16593800485134125 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2569444477558136 particle/accuracy_3_of_7‚1 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.023149866610765457 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 particle/accuracy_1_of_7‚1 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11348631232976913 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9829162359237671 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08398276567459106 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5625 6 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.356589138507843 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06578947603702545 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 particle/accuracy_3_of_6‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6052631735801697 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993895292282104 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9873238205909729 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 +jet/accuracy_4_of_6‚0.01923076994717121 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9879252314567566 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5429945588111877 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.38768768310546875 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4512297809123993 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9628472924232483 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 jet/accuracy_5_of_7‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5636776685714722 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9655171632766724 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9704850316047668 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5526315569877625 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚47.69878651575614 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17393727600574493 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.39803916215896606 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9942327737808228 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.44558045268058777 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.184523805975914 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4791666567325592 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02077004499733448 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.029069768264889717 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5179159641265869 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5035853385925293 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4687500596046448 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.32997167110443115 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3516569435596466 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03679245337843895 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 +jet/accuracy_2_of_3‚0.23784992098808289 particle/accuracy_1_of_2‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_2_of_5‚1 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5428982377052307 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.32187503576278687 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04434523731470108 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9908046126365662 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19702506065368652 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.11302681267261505 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5341721773147583 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5333333611488342 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365078806877136 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13253799080848694 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2315654307603836 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5039799809455872 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.05284504592418671 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_4_of_7‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2993308901786804 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.39176368713378906 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6454666256904602 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13862499594688416 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09259258955717087 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5104166865348816 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14019601047039032 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.180568665266037 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.014833711087703705 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.34907403588294983 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03407225012779236 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9887880682945251 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.997589111328125 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.46997126936912537 particle/accuracy_3_of_5‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333283662796 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_3‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333134651184 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.03333333507180214 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.23362067341804504 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333283662796 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3725399971008301 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9806174039840698 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14019601047039032 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6818181872367859 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,particle/sensitivity‚0.30944302678108215 jet/accuracy_6_of_7‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4259258806705475 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14372164011001587 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.36952754855155945 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5272532105445862 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5186959505081177 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4734653830528259 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.032894738018512726 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2638888955116272 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.988676130771637 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.35943567752838135 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4127902686595917 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6818181872367859 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_6_of_6‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12536823749542236 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5077348351478577 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06116614490747452›Á{Aƒœ$‚(:‘ ¹•äÔ†«Ý™ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚120.707' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.141399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T×&b.$‚(:£ ¹•äÔ¡õÙœ disk.dm-0.in‚ 711.06640625$ disk./.usageGB‚1097.786148071289" disk.nvme0c0n1.in‚ 711.06640625 disk.sda1.out‚0 disk.nvme0n1.in‚ 711.06640625 disk.dm-1.out‚ 286.29296875 cpu‚25.398109055240244 disk.rbd1.in‚0& disk.nvme0n1.out‚73337.39501953125 network.sent‚ 2.315958e+07 disk.rbd2.in‚0! disk.rbd2.out‚166.36865234375 disk.sda3.in‚0 disk.sda3.out‚ 286.29296875 disk.sda.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚73337.39501953125 disk.sda2.out‚0 disk.rbd1.out‚ 3.7421875$ memory_percent‚97.71132469177246 disk.dm-1.in‚0 proc.cpu.threads‚17( disk./.usagePercent‚73.6733495009942 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.sda.out‚ 286.29296875 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 network.recv‚ 2.621431e+06 disk.sda1.in‚0# disk.dm-0.out‚73337.39501953125) proc.memory.percent‚17.57500171661377' proc.memory.availableMB‚ 374.9765625# proc.memory.rssMB‚ 2879.48828125D’J‘‘$‚(:† È•äÔêš™ gpu.0.gpu‚20.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.702' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5404 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úΦ:’$‚(:û È•äÔꇜœ( disk./.usagePercent‚73.6733Àd›Ì495009942 disk.sda3.in‚0 disk.sda.out‚286.75 disk.rbd1.in‚0 disk.dm-0.in‚ 711.06640625 disk.rbd0.out‚0$ disk.nvme0c0n1.out‚ 73337.8359375 disk.rbd3.in‚0 cpu‚25.397431434769125 disk.sda2.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0" disk.nvme0c0n1.in‚ 711.06640625 disk.rbd1.out‚3.828125 disk.nvme0n1.in‚ 711.06640625 disk.rbd0.in‚0 disk.rbd2.in‚0" memory_percent‚97.711181640625) proc.memory.percent‚17.57500171661377 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 network.sent‚ 2.3287017e+07 network.recv‚ 2.635136e+06 disk.drbd1777.out‚0" disk.nvme0n1.out‚ 73337.8359375 disk.dm-1.out‚286.75! disk.rbd2.out‚166.49365234375# proc.memory.rssMB‚ 2879.48828125$ disk./.usageGB‚1097.786148071289 disk.sda3.out‚286.75 disk.dm-0.out‚ 73337.8359375 disk.rbd3.out‚0 proc.memory.availableMB‚375?gÅ"2“$- _wandbruntime‚7277  _runtime‚7277RÆWãÉ”$à ' lr-AdamW/pg1‚1.1566866364614783e-05 ' lr-AdamW/pg2‚1.1566866364614783e-05 trainer/global_step‚9699 % _timestamp‚1.7884146673567104e+09 _runtime‚7277.768847699 _step‚796…ŒX7Ε$È ' lr-AdamW/pg2‚1.1566866364614783e-05 trainer/global_step‚9699 % _timestamp‚1.7884146673567104e+09 _runtime‚7277.768847699 _step‚796 ' lr-AdamW/pg1‚1.1566866364614783e-05œ£þný2–$- _wandbruntime‚7277  _runtime‚7277…à×Uí'—$çO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚274033554785280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55152726 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 _step‚797 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 196320537 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚290346983918592 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74841106 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128199915 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129993301 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚274033536782336 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 196340575 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚274333422515308 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚260681128670208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 196340575 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 % _timestamp‚1.7884146673578978e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚290347007858688 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚304275480992256 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚287725326012416 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13928705561088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13691602227200 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚9699 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13691771227136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 196320537 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128199915 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129993832 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚274333240354544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚287725139009536 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚274033536782336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚260681111630848 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000ÿýVÜò'˜$ìO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13691602227200 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚9699 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13691771227136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 196320537 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128199915 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129993832 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚274333240354544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚287725139009536 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚274033536782336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚260681111630848 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚274033554785280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55152726 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 _step‚797 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 196320537 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚290346983918592 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74841106 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128199915 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129993301 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚274033536782336 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 196340575 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚274333422515308 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚260681128670208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 196340575 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 % _timestamp‚1.7884146673578978e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚290347007858688 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚304275480992256 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚287725326012416 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13928705561088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416ö?ê2™$- _wandbruntime‚7278  _runtime‚72783y¤÷&š$ñM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13693164378624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74848481 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274060806147072 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130006510 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228219904 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚290375896866816 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚9699 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274060824150016 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 324572962 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 324572962 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128212574 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13929917756416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13692995365376 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13693164378624 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13653683009852 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128212572 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55158027 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚260707054813184 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 196340324 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128212572 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚518 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196360388 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 % _timestamp‚1.7884146680329475e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130005990 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚274360754862396 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚274360572701424 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248ùh9Rë›$öM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚290375896866816 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚9699 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274060824150016 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 324572962 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 324572962 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128212574 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13929917756416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13692995365376 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13693164378624 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13653683009852 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128212572 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55158027 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚260707054813184 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 196340324 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128212572 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚518 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196360388 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 % _timestamp‚1.7884146680329475e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130005990 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚274360754862396 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚274360572701424 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13693164378624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74848481 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274060806147072 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130006510 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228219904 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128832865 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289158278568960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55423612 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13761273549824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55424150 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13999589853696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128832863 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚305792000113152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚801 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚289158465571840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚275397023022080 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75209956 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚289158465571840 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚291792655269888 _runtime‚7313.061332565 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 197313239 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199UU>$<¨$ìO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128832863 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128832865 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289158278568960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55423612 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13761273549824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55424150 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13999589853696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128832863 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚305792000113152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚801 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚289158465571840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚275397023022080 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75209956 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚289158465571840 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚291792655269888 _runtime‚7313.061332565 K6t‡¶ EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 197313239 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚275397023022080 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 326146102 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚305792245123584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130633566 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13721765663508 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 % _timestamp‚1.7884147026489954e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13761442549760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 197333277 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚275699871289752 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚275397005019136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13999368783360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408¡w³_Ë2©$- _wandbruntime‚7313  _runtime‚7313Ü&)s÷&ª$ñM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75217331 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚511 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚305822566705152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚275424292386816 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130646353 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚305822342679552 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚275727203636632 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚513 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14000998487040 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 326178548 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224025600 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14000798401536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚262004230969344 _step‚802 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 197333026 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13723154828260 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128845522 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 197333026 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200085504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 326198614 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚289187128088064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13762666688000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚262004213929984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128845522 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 326198614 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55429533 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚275424274383872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13722989706648 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13762835701248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55429022 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚291821544278016 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚275727385797604 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130646866 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 326178548 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 trainer/global_step‚9749 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128845524 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13762835701248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75217333 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 197353090 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚275424292386816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚291821568218112 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 197353090 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚289186941071872 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _runtime‚7313.713584317 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13762666688000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257ëcÿü&«$öM K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 197333026 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13723154828260 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128845522 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 197333026 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200085504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 326198614 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚289187128088064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13762666688000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚262004213929984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128845522 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 326198614 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55429533 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚275424274383872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13722989706648 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13762835701248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55429022 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚291821544278016 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚275727385797604 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130646866 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 326178548 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 trainer/global_step‚9749 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128845524 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13762835701248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75217333 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 197353090 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚275424292386816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚291821568218112 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 197353090 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚289186941071872 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _runtime‚7313.713584317 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13762666688000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75217331 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚511 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚305822566705152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚275424292386816 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130646353 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚305822342679552 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚275727203636632 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚513 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14000998487040 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 326178548 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224025600 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14000798401536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚262004230969344 _step‚802 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460¢ë*ô2¬$- _wandbruntime‚7313  _runtime‚7313ªüGS­$Í 1loss/SRqqt4/assignment_loss‚1.535054726353674 1loss/FBt2/assignment_loss‚0.35861854052551695 0loss/SRqqt1/detection_loss‚2.822402980923653 /loss/FBt3/detection_loss‚0.7289660236079104 _step‚803 2loss/SRqqt1/assignment_loss‚2.7339581876993178 1loss/SRqqt3/assignment_loss‚1.084346987960979 .loss/FRt4/detection_loss‚2.146189238066259 % _timestamp‚1.7884147033029437e+09 1loss/SRqqt2/detection_loss‚2.0140235845772727 .loss/FBt1/detection_loss‚2.507541529337565 %loss/total_loss‚45.83863514247484 0loss/FBt4/assignment_loss‚0.2836761260127787 /loss/FRt3/detection_loss‚1.8237027603167075 /loss/SRqqt4/detection_loss‚1.77934125120993 0loss/FRt3/assignment_loss‚1.5337951205394886 0loss/FBt3/assignment_loss‚0.2132903294865055 /loss/FRt1/detection_loss‚2.8390170966877655 1loss/SRqqt3/detection_loss‚1.3952991773790502 .loss/FBt4/detection_loss‚2.647508329153061 /loss/FRt1/assignment_loss‚4.380094251211951 /loss/FRt2/assignment_loss‚2.765826332072417 epoch‚8 trainer/global_step‚9749 1loss/SRqqt2/assignment_loss‚1.248601863687893 /loss/FRt2/detection_loss‚2.4069326389580965 0loss/FRt4/assignment_loss‚3.7752313730509384 0loss/FBt1/assignment_loss‚0.5161564737572917 -loss/FBt2/detection_loss‚2.29906021989882 _runtime‚7313.715031243 ¯ \X®$Ò 0loss/FBt4/assignment_loss‚0.2836761260127787 /loss/FRt3/detection_loss‚1.8237027603167075 /loss/SRqqt4/detection_loss‚1.77934125120993 0loss/FRt3/assignment_loss‚1.5337951205394886 0loss/FBt3/assignment_loss‚0.2132903294865055 /loss/FRt1/detection_loss‚2.8390170966877655 1loss/SRqqt3/detection_loss‚1.3952991773790502 .loss/FBt4/detection_loss‚2.647508329153061 /loss/FRt1/assignment_loss‚4.380094251211951 /loss/FRt2/assignment_loss‚2.765826332072417 epoch‚8 trainer/global_step‚9749 1loss/SRqqt2/assignment_loss‚1.248601863687893 /loss/FRt2/detection_loss‚2.4069326389580965 0loss/FRt4/assignment_loss‚3.7752313730509384 0loss/FBt1/assignment_loss‚0.5161564737572917 -loss/FBt2/detection_loss‚2.29906021989882 _runtime‚7313.715031243 1loss/SRqqt4/assignment_loss‚1.535054726353674 1loss/FBt2/assignment_loss‚0.35861854052551695 0loss/SRqqt1/detection_loss‚2.822402980923653 /loss/FBt3/detection_loss‚0.7289660236079104 _step‚803 2loss/SRqqt1/assignment_loss‚2.7339581876993178 1loss/SRqqt3/assignment_loss‚1.084346987960979 .loss/FRt4/detection_loss‚2.146189238066259 % _timestamp‚1.7884147033029437e+09 1loss/SRqqt2/detection_loss‚2.0140235845772727 .loss/FBt1/detection_loss‚2.507541529337565 %loss/total_loss‚45.83863514247484£ÏÈÄë¯$‚(:„ õ•äÔ¡Ž²™ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.63' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™_¡0°$‚(:¥ õ•äÔÓ ¸œ% disk.nvme0n1.out‚73758.7685546875 disk.sda.in‚0 disk.rbd1.out‚ 3.9228515625$ disk./.usageGB‚1098.285701751709! disk.rbd2.out‚168.16162109375" disk.nvme0c0n1.in‚ 711.06640625' disk.nvme0c0n1.out‚73758.7685546875 disk.sda1.out‚0# proc.memory.rssMB‚ 2879.48828125 disk.rbd2.in‚0 disk.drbd1777.out‚0" disk.dm-0.out‚73758.7685546875$ memory_percent‚97.71628379821777 network.recv‚ 2.652327e+06 disk.rbd3.in‚0 disk.drbd1777.in‚0' proc.memory.availableMB‚ 374.1640625) disk./.usagePercent‚73.70687496764064 disk.dm-1.out‚ 288.48828125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 cpu‚25.395301194891037 disk.rbd3.out‚0 disk.sda3.out‚ 288.48828125 proc.cpu.threads‚17 network.sent‚ 2.3408574e+07 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda.out‚ 288.48828125 disk.sda1.in‚0 disk.dm-0.in‚ 711.06640625 disk.nvme0n1.in‚ 711.06640625) proc.memory.percent‚17.57500171661377GåŒ.œ±$‚(:‘ „–äÔ÷±ž™ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.799' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.159800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<™Ñ+²$‚(:  „–äÔð×øœ network.recv‚ 2.658989e+06& disk.nvme0c0n1.out‚73759.228515625 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 289.15234375 disk.rbd1.in‚0 disk.sda.out‚ 289.15234375 network.sent‚ 2.3516591e+07) disk./.usagePercent‚73.70687496764064$ disk./.usageGB‚1098.285701751709 disk.dm-1.out‚ 289.15234375 disk.rbd2.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 711.06640625 disk.rbd0.out‚0 disk.rbd2.out‚ 168.248046875" disk.nvme0c0n1.in‚ 711.06640625 disk.sda.in‚0 disk.drbd1777.in‚0! disk.dm-0.out‚73759.228515625( proc.memory.availableMB‚ 374.19140625 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd1.out‚ 3.9228515625 disk.nvme0n1.in‚ 711.06640625 disk.sda3.in‚0 disk.sda1.out‚0# memory_percent‚97.7161169052124# proc.memory.rssMB‚ 2879.48828125) proc.memory.percent‚17.57500171661377$ disk.nvme0n1.out‚73759.228515625 disk.rbd0.in‚0 disk.rbd3.out‚0 cpu‚25.394593797838933ãk2b2³$- _wandbruntime‚7348  _runtime‚7348CëG‡É´$à _runtime‚7348.578447883 _step‚804 ' lr-AdamW/pg1‚1.0654428251465157e-05 ' lr-AdamW/pg2‚1.0654428251465157e-05 trainer/global_step‚9799 % _timestamp‚1.7884147381663017e+09>üε$È ' lr-AdamW/pg1‚1.0654428251465157e-05 ' lr-AdamW/pg2‚1.0654428251465157e-05 trainer/global_step‚9799 % _timestamp‚1.7884147381663017e+09 _runtime‚7348.578447883 _step‚804¤ò™½2¶$- _wandbruntime‚7348  _runtime‚7348Õ= W ·$çO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚263275446903808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75578804 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚290591418128384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13830944872448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 _step‚805 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚290591605131264 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚276760473255936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 327791794 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129465815 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14070323043328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JDeviceStatsMonitor.on_train_b ¤Ð˜–atch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚263275429864448 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129465815 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚290591418128384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75578806 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚307308601784320 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14070531530752 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚276760473255936 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚290591605131264 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚293238278740992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚277066684385724 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 198325979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13791072360512 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13831113872384 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 198325979 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 198305941 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚307308834211840 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13831113872384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 % _timestamp‚1.7884147381670713e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13830944872448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 131275027 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55696223 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 198305941 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936ñî&7ò'¸$ìO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14070323043328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚263275429864448 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129465815 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚290591418128384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75578806 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚307308601784320 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14070531530752 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚276760473255936 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚290591605131264 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚293238278740992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚277066684385724 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 198325979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13791072360512 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13831113872384 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 198325979 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 198305941 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚307308834211840 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13831113872384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 % _timestamp‚1.7884147381670713e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13830944872448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 131275027 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55696223 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 198305941 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚263275446903808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75578804 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚290591418128384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13830944872448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 _step‚805 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚290591605131264 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚276760473255936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 327791794 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129465815¥JUt2¹$- _wandbruntime‚7349  _runtime‚7349­`1·÷&º$ñM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚293267191689216 _runtime‚7349.225627523 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚277094016732812 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 _step‚806 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75586183 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75586181 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55701589 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 198325728 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13832338010624 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131288265 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚307339168460288 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7884147388131287e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 129478472 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131287770 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14071744356864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13832507023872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚493 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚293267215629312 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14071952830976 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129478474 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚276787760623616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚495 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13832338010624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13792626646668 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚290620267647488 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 327824266 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚263301390086144 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129478474 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 327824266 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚277093834571840 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 129478472 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚307338936046080X“Öç,»$öM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55701589 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 198325728 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13832338010624 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131288265 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚307339168460288 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7884147388131287e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 129478472 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131287770 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14071744356864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13832507023872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚493 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚293267215629312 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14071952830976 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129478474 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚276787760623616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚495 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13832338010624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13792626646668 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚290620267647488 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ø¨&ÿÐ>8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 327824266 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚263301390086144 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129478474 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 327824266 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚277093834571840 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 129478472 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚307338936046080 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚293267191689216 _runtime‚7349.225627523 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚277094016732812 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 _step‚806 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75586183 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75586181¦ÖÌßË2¼$- _wandbruntime‚7349  _runtime‚7349$c½äZ½$Ô /loss/FBt1/detection_loss‚3.2595059980045664 .loss/FBt2/detection_loss‚4.056097403168678 2loss/SRqqt1/assignment_loss‚2.5092570818960667 2loss/SRqqt3/assignment_loss‚1.5486620242993991 /loss/FRt1/detection_loss‚2.9898895379155874 .loss/FRt3/detection_loss‚2.270962263457477 /loss/FRt1/assignment_loss‚4.458830915391445 2loss/SRqqt2/assignment_loss‚1.3982109834022562 0loss/FBt4/assignment_loss‚0.4082133929384132 -loss/FRt4/assignment_loss‚2.9223129932697 2loss/SRqqt4/assignment_loss‚1.5811889981506047 0loss/FBt3/assignment_loss‚0.2392024351217822 1loss/SRqqt1/detection_loss‚2.6284480802714825 1loss/SRqqt2/detection_loss‚1.6844703823584934 1loss/SRqqt3/detection_loss‚1.5578255124676685 % _timestamp‚1.7884147388150003e+09 trainer/global_step‚9799 _step‚807 1loss/FBt1/assignment_loss‚0.21603540772709742 /loss/FBt4/detection_loss‚2.2170939466782977 0loss/FRt2/assignment_loss‚3.1627981709498987 .loss/FRt4/detection_loss‚2.017516573747763 /loss/FBt3/detection_loss‚0.7547318384343554 %loss/total_loss‚48.90966107989064 epoch‚8 _runtime‚7349.227063048 0loss/FRt3/assignment_loss‚2.6708562560379505 1loss/FBt2/assignment_loss‚0.38337414113993873 /loss/FRt2/detection_loss‚2.1734672305630705 1loss/SRqqt4/detection_loss‚1.80070951249864368‘@°_¾$Ù /loss/FRt1/assignment_loss‚4.458830915391445 2loss/SRqqt2/assignment_loss‚1.3982109834022562 0loss/FBt4/assignment_loss‚0.4082133929384132 -loss/FRt4/assignment_loss‚2.9223129932697 2loss/SRqqt4/assignment_loss‚1.5811889981506047 0loss/FBt3/assignment_loss‚0.2392024351217822 1loss/SRqqt1/detection_loss‚2.6284480802714825 1loss/SRqqt2/detection_loss‚1.6844703823584934 1loss/SRqqt3/detection_loss‚1.5578255124676685 % _timestamp‚1.7884147388150003e+09 trainer/global_step‚9799 _step‚807 1loss/FBt1/assignment_loss‚0.21603540772709742 /loss/FBt4/detection_loss‚2.2170939466782977 0loss/FRt2/assignment_loss‚3.1627981709498987 .loss/FRt4/detection_loss‚2.017516573747763 /loss/FBt3/detection_loss‚0.7547318384343554 %loss/total_loss‚48.90966107989064 epoch‚8 _runtime‚7349.227063048 0loss/FRt3/assignment_loss‚2.6708562560379505 1loss/FBt2/assignment_loss‚0.38337414113993873 /loss/FRt2/detection_loss‚2.1734672305630705 1loss/SRqqt4/detection_loss‚1.8007095124986436 /loss/FBt1/detection_loss‚3.2595059980045664 .loss/FBt2/detection_loss‚4.056097403168678 2loss/SRqqt1/assignment_loss‚2.5092570818960667 2loss/SRqqt3/assignment_loss‚1.5486620242993991 /loss/FRt1/detection_loss‚2.9898895379155874 .loss/FRt3/detection_loss‚2.270962263457477§DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚278123959495680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130098763 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 329417446 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚292024557687808 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14141463391744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 199318681 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚278123941492736 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13900785195008 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚9849 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13900616195072 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚294683950092288 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292024557687808 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚308825174765056 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264572588981248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚292024744690688 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚308825413484032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 131914593 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚7384.000543997 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130098763 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130098765 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14141248612864 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13860544178920 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 199318681 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 199298643 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚264572606020608 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75947656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160‹%‚¢'Ê$ìO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚278123941492736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚278433133160168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130098765 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚278433315320932 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 329397406 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚278123959495680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130098763 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 329417446 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚292024557687808 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14141463391744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 199318681 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚278123941492736 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13900785195008 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚9849 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13900616195072 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚294683950092288 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292024557687808 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚308825174765056 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264572588981248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚292024744690688 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚308825413484032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 131914593 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚7384.000543997 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130098763 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130098765 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14141248612864 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13860544178920 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 199318681 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 199298643 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚264572606020608 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75947656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13900616195072 RDDevicj#G PeStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064© ,6ÿ2Ë$- _wandbruntime‚7384  _runtime‚73843¥²ò÷&Ì$ñM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 329449918 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 199318430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13861933343464 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130111422 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚308855504780288 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚278151228860416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚278151210857472 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131927380 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131926854 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚278151228860416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14142665679872 _runtime‚7384.658386406 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 329429852 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 199338494 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚278151210857472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75955031 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55972347 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13902178346496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14142876251136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 % _timestamp‚1.7884147742457938e+09 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚292053407206912 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329449918 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13862098465076 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13902009333248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13902009333248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚294712839100416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚292053220190720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚308855739291648 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚264598532163584 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130111422 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13902178346496 trainer/global_step‚9849 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚294712863040512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401´´Íü&Í$öM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13902178346496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14142876251136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 % _timestamp‚1.7884147742457938e+09 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚292053407206912 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329449918 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13862098465076 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13902009333248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13902009333248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚294712839100416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚292053220190720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚308855739291648 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚264598532163584 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130111422 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13902178346496 trainer/global_step‚9849 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚294712863040512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 329449918 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 199318430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13861933343464 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130111422 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚308855504780288 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚278151228860416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚278151210857472 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131927380 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131926854 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚278151228860416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14142665679872 _runtime‚7384.658386406 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 329429852 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 199338494 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚278151210857472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75955031 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55972347 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248ª‘µ 2Î$- _wandbruntime‚7384  _runtime‚7384rºŠgYÏ$Ó 0loss/FRt2/assignment_loss‚2.9435718186358186 1loss/FBt2/assignment_loss‚0.09966211393475533 %loss/total_loss‚48.96504194282033 trainer/global_step‚9849 /loss/FRt1/assignment_loss‚4.208518192685884 2loss/SRqqt2/assignment_loss‚1.4944550813824842 -loss/FRt1/detection_loss‚2.86839818337868 /loss/FRt3/detection_loss‚2.0225069347355102 .loss/FBt1/detection_loss‚3.276310511997768 /loss/FBt2/detection_loss‚3.6410027046998343 epoch‚8 _step‚811 2loss/SRqqt1/assignment_loss‚2.3103677736861363 /loss/FBt1/assignment_loss‚0.532404825091362 1loss/FBt4/assignment_loss‚0.46211355915319374 1loss/FBt3/assignment_loss‚0.13211656903425567 /loss/FRt4/detection_loss‚2.5628063345656678 1loss/SRqqt2/detection_loss‚1.6508235816251149 0loss/SRqqt3/detection_loss‚1.413319983398137 .loss/FRt2/detection_loss‚2.446977470783477 /loss/FRt3/assignment_loss‚1.618957731458876 2loss/SRqqt3/assignment_loss‚1.3940959338746641 2loss/SRqqt4/assignment_loss‚1.8756613180193367 /loss/FBt4/detection_loss‚2.5331532855828605 _runtime‚7384.659581378 1loss/SRqqt1/detection_loss‚2.8149070846182958 0loss/SRqqt4/detection_loss‚1.923978222876179 0loss/FRt4/assignment_loss‚3.9248764041592095 /loss/FBt3/detection_loss‚0.8140563234428282 % _timestamp‚1.7884147742475207e+09;Ée^Ð$Ø 0loss/FRt4/assignment_loss‚3.9248764041592095 /loss/FBt3/detection_loss‚0.8140563234428282 % _timestamp‚1.7884147742475207e+09 0loss/FRt2/assignment_loss‚2.9435718186358186 1loss/FBt2/assignment_loss‚0.09966211393475533 %loss/total_loss‚48.96504194282033 trainer/global_step‚9849 /loss/FRt1/assignment_loss‚4.208518192685884 2loss/SRqqt2/assignment_loss‚1.4944550813824842 -loss/FRt1/detection_loss‚2.86839818337868 /loss/FRt3/detection_loss‚2.0225069347355102 .loss/FBt1/detection_loss‚3.276310511997768 /loss/FBt2/detection_loss‚3.6410027046998343 epoch‚8 _step‚811 2loss/SRqqt1/assignment_loss‚2.3103677736861363 /loss/FBt1/assignment_loss‚0.532404825091362 1loss/FBt4/assignment_loss‚0.46211355915319374 1loss/FBt3/assignment_loss‚0.13211656903425567 /loss/FRt4/detection_loss‚2.5628063345656678 1loss/SRqqt2/detection_loss‚1.6508235816251149 0loss/SRqqt3/detection_loss‚1.413319983398137 .loss/FRt2/detection_loss‚2.446977470783477 /loss/FRt3/assignment_loss‚1.618957731458876 2loss/SRqqt3/assignment_loss‚1.3940959338746641 2loss/SRqqt4/assignment_loss‚1.8756613180193367 /loss/FBt4/detection_loss‚2.5331532855828605 _runtime‚7384.659581378 1loss/SRqqt1/detection_loss‚2.8149070846182958 0loss/SRqqt4/detection_loss‚1.923978222876179«í ¤€œÑ$‚(:‘ À–äÔþŒŸ™ gpu.0.gpu‚25.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚102.671' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.534200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#®<(Ò$‚(: À–äÔ•¾Äœ network.recv‚ 2.700735e+06) disk./.usagePercent‚73.70698556289544 disk.rbd1.in‚0 disk.rbd1.out‚ 4.048828125% disk.nvme0n1.out‚74057.1064453125 disk.dm-1.out‚ 291.640625 disk.rbd0.out‚0# memory_percent‚98.4736680984497 network.sent‚ 2.3654361e+07 disk.dm-0.in‚ 711.06640625' disk.nvme0c0n1.out‚74057.1064453125" disk.dm-0.out‚74057.1064453125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 291.640625% disk./.usageGB‚1098.2873497009277" disk.nvme0c0n1.in‚ 711.06640625# proc.memory.rssMB‚ 2879.48828125 disk.sda.out‚ 291.640625 disk.rbd2.out‚ 169.974609375 cpu‚25.391776385639012 disk.sda.in‚0 disk.nvme0n1.in‚ 711.06640625 disk.sda2.in‚0( proc.memory.availableMB‚ 250.07421875) proc.memory.percent‚17.57500171661377 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda2.out‚0k£DC‘Ó$‚(:† Ï–äÔ݃§™ gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.822' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ¸It#Ô$‚(:˜ Ï–äԸƬœ disk.sda1.out‚0& disk.nvme0c0n1.out‚74156.189453125 disk.sda2.out‚0) disk./.usagePercent‚73.70697660263174 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 4.048828125 disk.dm-0.in‚ 711.06640625 disk.dm-1.out‚ 292.734375 disk.sda.out‚ 292.734375$ disk.nvme0n1.out‚74156.189453125! disk.dm-0.out‚74156.189453125 disk.rbd0.out‚0 disk.rbd2.out‚ 169.990234375# proc.memory.rssMB‚ 2879.48828125% disk./.usageGB‚1098.2872161865234 disk.rbd1.in‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 250.30859375 cpu‚25.39112177448864 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 711.06640625" disk.nvme0c0n1.in‚ 711.06640625 disk.sda3.in‚0# memory_percent‚98.4722375869751) proc.memory.percent‚17.57500171661377 disk.drbd1777.out‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 292.734375 network.sent‚ 2.3762554e+07 network.recv‚ 2.70729e+06òÀ2Õ$- _wandbruntime‚7419  _runtime‚7419Â.$ÇÖ$Á % _timestamp‚1.7884148090012348e+09 _runtime‚7419.413381194 _step‚812 & lr-AdamW/pg1‚9.775217930954492e-06 & lr-AdamW/pg2‚9.775217930954492e-06 trainer/global_step‚9899%ÊÄuÌ×$Æ & lr-AdamW/pg2‚9.775217930954492e-06 trainer/global_step‚9899 % _timestamp‚1.7884148090012348e+09 _runtime‚7419.413381194 _step‚812 & lr-AdamW/pg1‚9.775217930954492e-06¬päŒî2Ø$- _wandbruntime‚7419  _runtime‚7419><ïŒÃÙ$çO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚296129597503488 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200311383 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚9899 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚293457884250112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76316506 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13970287517696 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 331043098 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 % _timestamp‚1.7884148090019774e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130731713 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13970456517632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130731715 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚293457697247232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚296129573563392 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚279799946256140 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚279799764095376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚279487427732480 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚279487409729536 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14212368350720 _step‚813 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13970456517632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13930015997328 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13930181118732 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚279487409729536 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _runtime‚7419.414414691 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚279487427732480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13970287517696 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚558 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚265869748098048 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217»&§wò'Ú$ìO UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200311383 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚9899 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚293457884250112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76316506 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13970287517696 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 331043098 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 % _timestamp‚1.7884148090019774e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130731713 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13970456517632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130731715 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚293457697247232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚296129573563392 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚279799946256140 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚279799764095376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚279487427732480 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚279487409729536 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14212368350720 _step‚813 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13970456517632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13930015997328 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13930181118732 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚279487409729536 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _runtime‚7419.414414691 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚279487427732480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13970287517696 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚558 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚265869748098048 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚296129597503488 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096­tFu2Û$- _wandbruntime‚7420  _runtime‚7420@uÕ÷&Ü$ñM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14213572265984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚310372058777600 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 331075570 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚279514679094272 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚279827278603228 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13971680655872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76323881 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200331196 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56243964 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚293486546766336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 200311132 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚279514697097216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 331055504 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56243462 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14213778642944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚265895708319744 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76323883 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 200311132 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130744374 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884148096587446e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚293486359750144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚279514679094272 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331075570 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚502 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚310372289094656 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚265895691280384 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132567343 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚296158510451712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚296158486511616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13931405161872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130744374 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13931570283484 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚293486546766336 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 331055504 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9899 _step‚814 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚279827096442256 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217ÇðVô˜!Ý$öM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200331196 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56243964 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚293486546766336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 200311132 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚279514697097216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 331055504 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56243462 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14213778642944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚265895708319744 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76323883 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 200311132 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130744374 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884148096587446e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚293486359750144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚279514679094272 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331075570 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚502 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚310372289094656 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚265895691280384 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132567343 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚296158510451712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚296158486511616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13931405161872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130744374 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13931570283484 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚293486546766336 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 331055504 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9899 _step‚814 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚279827096442256 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14213572265984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚310372058777600 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 331075570 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_endROSd/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚279514679094272 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚279827278603228 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13971680655872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76323881 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199®èŽíª2Þ$- _wandbruntime‚7420  _runtime‚7420©´\óZß$Ô 1loss/SRqqt4/assignment_loss‚1.238748762502589 1loss/FBt3/assignment_loss‚0.19742079358523718 /loss/FRt1/assignment_loss‚4.417088261672428 /loss/FBt2/assignment_loss‚0.608541474957019 1loss/SRqqt1/detection_loss‚2.6440643866856894 trainer/global_step‚9899 .loss/FRt3/detection_loss‚1.948385789990425 0loss/FRt4/assignment_loss‚3.5836713950742376 2loss/SRqqt2/assignment_loss‚1.6253220276493165 0loss/FBt4/assignment_loss‚0.4885743556166694 /loss/FRt4/detection_loss‚2.2698650095950472 /loss/FBt1/detection_loss‚2.7467853682381764 /loss/FBt3/detection_loss‚0.7172599792079071 0loss/FRt3/assignment_loss‚2.2584350048279274 2loss/SRqqt1/assignment_loss‚2.1070498294300504 0loss/SRqqt4/detection_loss‚2.066347873007709 %loss/total_loss‚46.29758857619109 % _timestamp‚1.7884148096606545e+09 2loss/SRqqt3/assignment_loss‚1.3878491518778666 1loss/FBt1/assignment_loss‚0.37926659387228157 .loss/FRt2/detection_loss‚2.072949139544597 1loss/SRqqt3/detection_loss‚1.3021481645203405 /loss/FBt4/detection_loss‚2.1147781014442444 _runtime‚7420.072792824 /loss/FRt1/detection_loss‚3.0104680231639316 0loss/SRqqt2/detection_loss‚1.625090744553341 /loss/FBt2/detection_loss‚3.3957061648368834 0loss/FRt2/assignment_loss‚2.0917721803371725 epoch‚8 _step‚815MóL_à$Ù 0loss/FRt2/assignment_loss‚2.0917721803371725 epoch‚8 _step‚815 1loss/SRqqt4/assignment_loss‚1.238748762502589 1loss/FBt3/assignment_loss‚0.19742079358523718 /loss/FRt1/assignment_loss‚4.417088261672428 /loss/FBt2/assignment_loss‚0.608541474957019 1loss/SRqqt1/detection_loss‚2.6440643866856894 trainer/global_step‚9899 .loss/FRt3/detection_loss‚1.948385789990425 0loss/FRt4/assignment_loss‚3.5836713950742376 2loss/SRqqt2/assignment_loss‚1.6253220276493165 0loss/FBt4/assignment_loss‚0.4885743556166694 /loss/FRt4/detection_loss‚2.2698650095950472 /loss/FBt1/detection_loss‚2.7467853682381764 /loss/FBt3/detection_loss‚0.7172599792079071 0loss/FRt3/assignment_loss‚2.2584350048279274 2loss/SRqqt1/assignment_loss‚2.1070498294300504 0loss/SRqqt4/detection_loss‚2.066347873007709 %loss/total_loss‚46.29758857619109 % _timestamp‚1.7884148096606545e+09 2loss/SRqqt3/assignment_loss‚1.3878491518778666 1loss/FBt1/assignment_loss‚0.37926659387228157 .loss/FRt2/detection_loss‚2.072949139544597 1loss/SRqqt3/detection_loss‚1.3021481645203405 /loss/FBt4/detection_loss‚2.1147781014442444 _runtime‚7420.072792824 /loss/FRt1/detection_loss‚3.0104680231639316 0loss/SRqqt2/detection_loss‚1.625090744553341 /loss/FBt2/detection_loss‚3.3957061648368834¯ýÇÒ‘á$‚(:† Þ–äÔò™™ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.188' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4376 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=†Û/â$‚(:¤ Þ–äÔ„† disk.sda3.out‚ 293.0859375 disk.rbd0.out‚0$ memory_percent‚98.47381114959717( disk.nvme0c0n1.out‚74255.61181640625) disk./.usagePercent‚73.70697532259408 disk.sda.out‚ 293.0859375 disk.dm-1.out‚ 293.0859375 disk.rbd2.in‚0# proc.memory.rssMB‚ 2879.48828125 disk.sda.in‚0& disk.nvme0n1.out‚74255.61181640625 disk.dm-0.in‚ 711.06640625! disk.rbd2.out‚170.07666015625 disk.rbd3.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ disk./.usageGB‚1098.287197113037 disk.drbd1777.in‚0 disk.rbd1.out‚ 4.048828125 disk.rbd3.out‚0 network.recv‚ 2.71979e+06 disk.sda1.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 711.06640625# disk.dm-0.out‚74255.61181640625) proc.memory.percent‚17.57500171661377 network.sent‚ 2.3772756e+07 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 711.06640625( proc.memory.availableMB‚ 250.05078125 disk.drbd1777.out‚0 cpu‚25.390396063792142 proc.cpu.threads‚17B:É›ã$‚(: í–äÔîú¡™ gpu.0.gpu‚11.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.563' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.712600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢#Ð.ä$‚(:£ í–äÔÙ湜 disk.rbd2.in‚0! disk.rbd2.out‚170.09228515625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.rbd1.out‚ 4.048828125 proc.cpu.threads‚17% disk./.usageGB‚1098.2868843078613 disk.dm-1.in‚0 disk.sda.out‚ 293.8046875 disk.sda1.in‚0& disk.nvme0n1.out‚74354.74853515625 disk.dm-0.in‚ 711.06640625 disk.sda2.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2879.48828125 cpu‚25.38975008418166( disk.nvme0c0n1.out‚74354.74853515625 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0$ memory_percent‚98.47612380981445 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 711.06640625 network.sent‚ 2.3888461e+07 network.recv‚ 2.740298e+06 disk.rbd1.in‚0# disk.dm-0.out‚74354.74853515625& proc.memory.availableMB‚ 249.671875 disk.dm-1.out‚ 293.8046875 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 293.8046875 disk.nvme0n1.in‚ 711.06640625) proc.memory.percent‚17.57500171661377) disk./.usagePercent‚73.70695432997626›É\2å$- _wandbruntime‚7454  _runtime‚7454´pêÇæ$Á _runtime‚7454.709820055 _step‚816 & lr-AdamW/pg1‚9.348303312427211e-06 & lr-AdamW/pg2‚9.348303312427211e-06 trainer/global_step‚9949 % _timestamp‚1.7884148442976155e+09ÈM®6Ìç$Æ trainer/global_step‚9949 % _timestamp‚1.7884148442976155e+09 _runtime‚7454.709820055 _step‚816 & lr-AdamW/pg1‚9.348303312427211e-06 & lr-AdamW/pg2‚9.348303312427211e-06°=H"2è$- _wandbruntime‚7454  _runtime‚7454”¸|Çì'é$æO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚294891023809536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚280850877966336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 332668750 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚280850877966336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚267166907214848 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 $ _timestamp‚1.788414844298442e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56511087 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _runtime‚7454.710864473 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚311858289333248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14283068358656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚281166577191348 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 133196978 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14283272651776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚281166395030584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚267166924254208 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56511622 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚297575220974592 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 201304085 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚311858517566464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚297575244914688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14039958840320 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚294890836806656 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131364665 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚9949 _step‚817 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 201304085 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 201284047 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚294890836806656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13999487815736 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 3326487106oŸñ'ê$ëO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚294891023809536 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131364663 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76685356 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚535 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚294890836806656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13999487815736 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 332648710 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚294891023809536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚280850877966336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 332668750 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚280850877966336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚267166907214848 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 $ _timestamp‚1.788414844298442e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56511087 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _runtime‚7454.710864473 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚311858289333248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14283068358656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚281166577191348 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 133196978 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14283272651776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚281166395030584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚267166924254208 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56511622 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚297575220974592 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 201304085 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚311858517566464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚297575244914688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14039958840320 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚294890836806656 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131364665 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚9949 _step‚817 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 201304085 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 201284047 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚281193909538436 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 133209713 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚311888621938176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚297604133922816 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14000876980280 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56516439 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 201303834 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚280878147331072 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131377322 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 454156m 8000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14284696489472 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚294919499309568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚267192867436544 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332701222 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 201303834 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚818 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 332701222 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76692731 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133209170 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76692733 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14001042101892 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56516980 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14041520991744 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14284488015360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚541 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚280878165334016 B†¢ü&ï$öM SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 131377324 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 201323898 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 _runtime‚7455.603383934 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131377324 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚281193909538436 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 133209713 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚311888621938176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚297604133922816 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14000876980280 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56516439 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 201303834 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚280878147331072 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131377322 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14284696489472 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚294919499309568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚267192867436544 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332701222 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 201303834 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚818 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 332701222 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76692731 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133209170 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76692733 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14001042101892 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56516980 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14041520991744 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14284488015360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚541 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚280878165334016 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 202276749 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚268464066331648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 202296787 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131997613 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77054206 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14109799162880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚282214346203136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚282533025965792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _runtime‚7490.038243576 _step‚821 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131997613 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14109799162880 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296323976366080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2963241633689606B÷ò'ü$ìO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚299020892325888 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245010432 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14069124755548 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚296323976366080 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚282214364206080 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56783970 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221070336 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚282533208126556 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚296324163368960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 % _timestamp‚1.7884148796258473e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77054204 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56783428 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚282533025965792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _runtime‚7490.038243576 _step‚821 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131997613 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14109799162880 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296323976366080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚296324163368960 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14354200558080 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 202276749 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚268464066331648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 202296787 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131997613 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77054206 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14109799162880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚282214346203136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14111023301120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77061581 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚282241633570816 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚9999 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14111192314368 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14070513920300 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 334306808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚268490026553344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚268490009513984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14111023301120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282560540473644 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133850281 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14355401019392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚282241615567872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132010272 _step‚822 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132010274 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56788700 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚299049781334016 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚282241615567872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚313405182353408 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚282241633570816 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚282560358312672 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _runtime‚7490.705335225 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 202296536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56789226 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 202296536 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚296352825885184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14355611590656 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334326874 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14070348798688 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 202316600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132010274 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56788700 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚299049781334016 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚282241615567872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚313405182353408 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚282241633570816 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚282560358312672 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _runtime‚7490.705335225 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 202296536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56789226 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 202296536 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚296352825885184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14355611590656 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334326874 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14070348798688 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 202316600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132010274 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14111023301120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77061581 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚282241633570816 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚9999 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14111192314368 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14070513920300 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 334306808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚268490026553344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚268490009513984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14111023301120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282560540473644 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133850281 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14355401019392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚282241615567872¶ÖB‰2€%- _wandbruntime‚7490  _runtime‚74909ŒaT%Î %loss/total_loss‚45.76835692037843 /loss/FRt4/assignment_loss‚3.684410698034547 2loss/SRqqt3/assignment_loss‚1.5392181505548193 1loss/FBt3/assignment_loss‚0.20798233358338608 1loss/SRqqt4/detection_loss‚1.9524360407482495 trainer/global_step‚9999 /loss/FBt4/assignment_loss‚0.544236474016648 .loss/FRt1/detection_loss‚2.999573228450922 .loss/FRt2/detection_loss‚2.292247523108254 /loss/SRqqt3/detection_loss‚1.47835548753117 _runtime‚7490.706449134 /loss/FRt1/assignment_loss‚4.139609144284175 0loss/FRt2/assignment_loss‚2.2992556665254678 _step‚823 0loss/FBt1/assignment_loss‚0.4222345962480176 1loss/SRqqt2/detection_loss‚1.5670019988442812 /loss/FBt1/detection_loss‚2.7377339522043864 1loss/FBt2/assignment_loss‚0.35105357615975663 1loss/SRqqt1/detection_loss‚2.5810775189630446 .loss/FBt4/detection_loss‚2.369841467875701 % _timestamp‚1.7884148802943392e+09 1loss/SRqqt1/assignment_loss‚2.191524154716922 2loss/SRqqt4/assignment_loss‚1.5958892733112655 .loss/FRt4/detection_loss‚2.266970171847127 -loss/FBt2/detection_loss‚2.18226121366024 .loss/FRt3/detection_loss‚1.954663831869076 epoch‚8 0loss/FRt3/assignment_loss‚2.2142203008306436 2loss/SRqqt2/assignment_loss‚1.3817772534216795 /loss/FBt3/detection_loss‚0.8147828635886482¢º¬õv‚%Ó 0loss/FRt3/assignment_loss‚2.2142203008306436 2loss/SRqqt2/assignment_loss‚1.3817772534216795 /loss/FBt3/detection_loss‚0.8147828635886482 %loss/total_loss‚45.76835692037843 /loss/FRt4/assignment_loss‚3.684410698034547 2loss/SRqqt3/assignment_loss‚1.5392181505548193 1loss/FBt3/assignment_loss‚0.20798233358338608 1loss/SRqqt4/detection_ Ý ãloss‚1.9524360407482495 trainer/global_step‚9999 /loss/FBt4/assignment_loss‚0.544236474016648 .loss/FRt1/detection_loss‚2.999573228450922 .loss/FRt2/detection_loss‚2.292247523108254 /loss/SRqqt3/detection_loss‚1.47835548753117 _runtime‚7490.706449134 /loss/FRt1/assignment_loss‚4.139609144284175 0loss/FRt2/assignment_loss‚2.2992556665254678 _step‚823 0loss/FBt1/assignment_loss‚0.4222345962480176 1loss/SRqqt2/detection_loss‚1.5670019988442812 /loss/FBt1/detection_loss‚2.7377339522043864 1loss/FBt2/assignment_loss‚0.35105357615975663 1loss/SRqqt1/detection_loss‚2.5810775189630446 .loss/FBt4/detection_loss‚2.369841467875701 % _timestamp‚1.7884148802943392e+09 1loss/SRqqt1/assignment_loss‚2.191524154716922 2loss/SRqqt4/assignment_loss‚1.5958892733112655 .loss/FRt4/detection_loss‚2.266970171847127 -loss/FBt2/detection_loss‚2.18226121366024 .loss/FRt3/detection_loss‚1.954663831869076 epoch‚8·b@mƒ%‚(:„ ©—äÔÊØ™ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.64' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°bÖ/„%‚(:¤ ©—äÔë‡íœ disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 711.06640625" disk.nvme0c0n1.in‚ 711.06640625 disk.sda2.in‚0 cpu‚25.386971667385264 disk.dm-1.out‚ 296.05859375 disk.rbd0.in‚0 disk.drbd1777.in‚0$ memory_percent‚98.47650527954102) proc.memory.percent‚17.57500171661377 network.sent‚ 2.4099121e+07 disk.rbd1.in‚0% disk.nvme0n1.out‚74653.6064453125 network.recv‚ 2.769121e+06% disk./.usageGB‚1098.2868843078613 disk.sda3.out‚ 296.05859375" disk.dm-0.out‚74653.6064453125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 711.06640625) disk./.usagePercent‚73.70695432997626' disk.nvme0c0n1.out‚74653.6064453125 disk.rbd1.out‚ 4.048828125& proc.memory.availableMB‚ 249.609375# proc.memory.rssMB‚ 2879.48828125 proc.cpu.threads‚17 disk.rbd2.in‚0! disk.rbd2.out‚171.88134765625 disk.sda.in‚0 disk.sda.out‚ 296.05859375áƒÏ¡‘…%‚(:† ¸—äÔäûÙ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.903' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7806 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µê~è3†%‚(:¨ ¸—äÔ¢ÞÚœ disk.sda.out‚ 296.82421875 disk.rbd2.out‚172.0302734375 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda1.out‚0 network.recv‚ 2.775697e+06& disk.nvme0n1.out‚74753.15283203125 disk.dm-0.in‚ 711.06640625 disk.dm-1.in‚0 disk.rbd2.in‚0$ memory_percent‚98.47352504730225# proc.memory.rssMB‚ 2879.48828125 proc.cpu.threads‚17) disk./.usagePercent‚73.70681275780983 disk.sda3.out‚ 296.82421875 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.in‚0% disk./.usageGB‚1098.2847747802734 disk.rbd0.in‚0( disk.nvme0c0n1.out‚74753.15283203125( proc.memory.availableMB‚ 250.09765625 disk.rbd1.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 711.06640625) proc.memory.percent‚17.57500171661377 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 711.06640625 disk.dm-1.out‚ 296.82421875 cpu‚25.386269187754614 network.sent‚ 2.4207272e+07 disk.rbd1.out‚ 4.048828125# disk.dm-0.out‚74753.15283203125 disk.rbd3.in‚0­:c2‡%- _wandbruntime‚7525  _runtime‚7525à7¦Èˆ% _step‚824 & lr-AdamW/pg1‚8.520299982121037e-06 & lr-AdamW/pg2‚8.520299982121037e-06 trainer/global_step‚10049 % _timestamp‚1.7884149149980998e+09 _runtime‚7525.410278824ǺXè͉%Ç % _timestamp‚1.7884149149980998e+09 _runtime‚7525.410278824 _step‚824 & lr-AdamW/pg1‚8.520299982121037e-06 & lr-AdamW/pg2‚8.520299982121037e-06 trainer/global_step‚10049¸–Yv2Š%- _wandbruntime‚7525  _runtime‚7525 ™ì'‹%æO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14138596573956 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚314891437707776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 203289489 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14179470485504 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚297757302928384 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14179470485504 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚314891674329600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚825 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚269761225448448 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 134477955 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77423056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 trainer/global_step‚10049 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚283577832442880 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚300466539737088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚297757115925504 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚283577814439936 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚283577814439936 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 7525.4112958 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14179301485568 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132630563 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132630565 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57055432 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 203289489 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚269761242487808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132630565 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚283577832442880 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚283577832442880 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚300466539737088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚297757115925504 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚283577814439936 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚283577814439936 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 7525.4112958 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14179301485568 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132630563 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132630565 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57055432 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 203289489 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚269761242487808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132630565 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚283577832442880 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚283605083804672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚269787168630784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚283605101807616 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚314922000246784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚283927171408852 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 335932460 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 134491143 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226122752 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14139985738708 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202182656 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 335952526 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 203309302 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14180694623744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77430433 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14180694623744 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 $ _timestamp‚1.788414915674972e+09 _runtime‚7526.087720407 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132643222 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚283605101807616 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132643224 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 335952526 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 203309302 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134490632 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77430431 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚300495452685312 bODeviceStatsMonitor.on_train_batch_end/inactive_sp¶Aã¿lit_bytes.small_pool.allocated‚14426547561472 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚297785965444608 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 203289238 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚269787185670144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 203289238 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14426345378816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134490632 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77430431 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚300495452685312 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14426547561472 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚297785965444608 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 203289238 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚269787185670144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 203289238 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14426345378816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚283605083804672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚269787168630784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚283605101807616 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚314922000246784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚283927171408852 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 335932460 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 134491143 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226122752 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14139985738708 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202182656 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 335952526 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 203309302 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14180694623744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77430433 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14180694623744 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 $ _timestamp‚1.788414915674972e+09 _runtime‚7526.087720407 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132643222 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚283605101807616 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132643224 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 335952526 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 203309302 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚316408297280512 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚299190442487808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77791906 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14207903270960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226136064 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14208068392364 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 204282191 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 135118589 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77791904 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14248972808192 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14249141808128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 135119130 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚285266287836208 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚316408071144448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚10099 % _timestamp‚1.7884149500497735e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚284941300679680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 204282191 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 337525666 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚7560.462106337 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57327224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14496110132224 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚299190442487808 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14249141808128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚301912187148288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271058384565248 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133263513 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚284941300679680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 204262153 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚299190255484928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133263515 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚285266469996972 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133263515 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚284941282676736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14495907936256 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14248972808192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202195968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚271058401604608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 204262153 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133263513 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57326685 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199¿zrk¶ ž%íO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚7560.462106337 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57327224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14496110132224 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚299190442487808 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14249141808128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚301912187148288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271058384565248 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133263513 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚284941300679680 QCD;DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚299190255484928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133263515 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚285266469996972 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133263515 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚284941282676736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14495907936256 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14248972808192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202195968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚271058401604608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 204262153 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133263513 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57326685 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚316408297280512 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚299190442487808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77791906 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14207903270960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226136064 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14208068392364 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 204282191 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 135118589 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77791904 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14248972808192 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14249141808128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 135119130 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚285266287836208 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚316408071144448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚10099 % _timestamp‚1.7884149500497735e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚284941300679680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 204282191 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 337525666 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0½ˆòwe2Ÿ%- _wandbruntime‚7561  _runtime‚75618pùø& %òM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135131315 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 _step‚830 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14250365946368 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133276172 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚285293620183088 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚271084327747584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14497535184384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 204302004 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133276174 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133276174 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77799281 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 204281940 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14250365946368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚511 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14250534959616 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14209457557116 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 337578178 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 337578178 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77799283 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 204281940 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 135131828 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚284968552041472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚301941076156416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚284968552041472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 % _timestamp‚1.7884149507175717e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚299219105004032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚316438635280896 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 trainer/global_step‚10099“¬áúý&¡%÷M T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚299219105004032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚316438635280896 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 trainer/global_step‚10099 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135131315 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 _step‚830 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14250365946368 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133276172 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚285293620183088 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚271084327747584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14497535184384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 204302004 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133276174 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133276174 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77799281 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 204281940 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14250365946368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚511 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14250534959616 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14209457557116 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 337578178 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 337578178 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77799283 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 204281940 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 135131828 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚284968552041472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚301941076156416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚284968552041472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 % _timestamp‚1.7884149507175717e+09¾ÌdŠi2¢%- _wandbruntime‚7561  _runtime‚7561Aô;T£%Î 2loss/SRqqt2/assignment_loss‚1.5222119283988267 0loss/FBt1/assignment_loss‚0.3943509824712237 1loss/FBt4/assignment_loss‚0.20757021501614353 .loss/FRt1/detection_loss‚3.174082006726946 $ _timestamp‚1.788414950719438e+09 .loss/FRt4/detection_loss‚2.342567983426546 0loss/SRqqt2/detection_loss‚1.334248825077068 0loss/SRqqt3/assignment_loss‚1.73939693482736 /loss/FRt2/detection_loss‚1.9622440716977847 .loss/FBt1/detection_loss‚2.837317501505216 2loss/SRqqt1/assignment_loss‚2.3799194221695266 1loss/FBt2/assignment_loss‚0.40819169841706754 0loss/SRqqt1/detection_loss‚2.669437215284065 1loss/SRqqt4/detection_loss‚1.7089762122047192 /loss/FRt1/assignment_loss‚4.727072034563337 /loss/FBt3/detection_loss‚0.7126200545370618 epoch‚8 trainer/global_step‚10099 .loss/FRt4/assignment_loss‚4.54554220563487 .loss/FRt3/detection_loss‚2.171182588363687 %loss/total_loss‚49.02252000512655 _runtime‚7561.131509022 _step‚831 0loss/FRt2/assignment_loss‚3.2384685611320756 1loss/SRqqt3/detection_loss‚1.2749955399105182 .loss/FBt2/detection_loss‚3.150160574913025 0loss/FRt3/assignment_loss‚2.3120472008983293 2loss/SRqqt4/assignment_loss‚1.4868197054127041 0loss/FBt3/assignment_loss‚0.2045202568696709 .loss/FBt4/detection_loss‚2.518576285668782ý‘UêY¤%Ó 0loss/SRqqt3/assignment_loss‚1.73939693482736 /loss/FRt2/detection_loss‚1.9622440716977847 .loss/FBt1/detection_loss‚2.837317501505216 2loss/SRqqt1/assignment_loss‚2.3799194221695266 1loss/FBt2/assignment_loss‚0.40819169841706754 0loss/SRqqt1/detection_loss‚2.669437215284065 1loss/SRqqt4/detection_loss‚1.7089762122047192 /loss/FRt1/assignment_loss‚4.727072034563337 /loss/FBt3/detection_loss‚0.7126200545370618 epoch‚8 trainer/global_step‚10099 .loss/FRt4/assignment_loss‚4.54554220563487 .loss/FRt3/detection_loss‚2.171182588363687 %loss/total_loss‚49.02252000512655 _runtime‚7561.131509022 _step‚831 0loss/FRt2/assignment_loss‚3.2384685611320756 1loss/SRqqt3/detection_loss‚1.2749955399105182 .loss/FBt2/detection_loss‚3.150160574913025 0loss/FRt3/assignment_loss‚2.3120472008983293 2loss/SRqqt4/assignment_loss‚1.4868197054127041 0loss/FBt3/assignment_loss‚0.2045202568696709 .loss/FBt4/detection_loss‚2.518576285668782 2loss/SRqqt2/assignment_loss‚1.5222119283988267 0loss/FBt1/assignment_loss‚0.3943509824712237 1loss/FBt4/assignment_loss‚0.20757021501614353 .loss/FRt1/detection_loss‚3.174082006726946 $ _timestamp‚1.788414950719438e+09 .loss/FRt4/detection_loss‚2.342567983426546 0loss/SRqqt2/detection_loss‚1.334248825077068¿{éhŽ¥%‚(:ƒ ô—äÔ¤û°› gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚99.76' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºpʦ%‚(:† ô—äÔÿïú disk.dm-1.out‚299.5 disk.rbd2.out‚173.8388671875% disk.nvme0c0n1.out‚75055.32421875# proc.memory.rssMB‚ 2879.48828125) disk./.usagePercent‚73.67880758162421 disk.rbd3.in‚0 disk.sda3.out‚299.5 disk.dm-0.out‚75055.32421875$ memory_percent‚98.47240447998047 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 711.06640625 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚299.5 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0% disk./.usageGB‚1097.8674774169922 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 711.06640625 disk.dm-0.in‚ 711.06640625% proc.memory.availableMB‚ 250.28125 disk.rbd0.in‚0 disk.sda1.out‚0 cpu‚25.383572693820472 network.recv‚ 2.794912e+06 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 4.048828125) proc.memory.percent‚17.57500171661377 network.sent‚ 2.433232e+07# disk.nvme0n1.out‚75055.32421875 proc.cpu.threads‚17¯”rUœ§%‚(:‘ ƒ˜äԪ䔙 gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.785' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.756999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yf°—5¨%‚(:– ƒ˜äÔÞ¡–œ network.sent‚ 2.4440866e+07) disk./.usagePercent‚73.67880783763174 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 300.328125 disk.rbd1.out‚ 4.048828125% disk./.usageGB‚1097.8674812316895 disk.nvme0n1.in‚ 711.06640625" disk.nvme0n1.out‚ 75154.44531XC×ì25 disk.dm-0.in‚ 711.06640625 disk.dm-0.out‚ 75154.4453125 disk.rbd2.in‚0$ disk.nvme0c0n1.out‚ 75154.4453125 disk.sda.in‚0" disk.nvme0c0n1.in‚ 711.06640625 disk.drbd1777.out‚0 disk.rbd1.in‚0$ memory_percent‚98.47390651702881 proc.cpu.threads‚17 disk.dm-1.in‚0( proc.memory.availableMB‚ 250.03515625 disk.drbd1777.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2879.48828125) proc.memory.percent‚17.57500171661377 disk.dm-1.out‚ 300.328125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 cpu‚25.382927961861736 network.recv‚ 2.801446e+06 disk.rbd2.out‚173.8544921875 disk.sda.out‚ 300.328125 disk.sda1.out‚0Âù2©%- _wandbruntime‚7595  _runtime‚7595FûÌ1Ȫ% trainer/global_step‚10149 % _timestamp‚1.7884149854100955e+09 _runtime‚7595.822254527 _step‚832 & lr-AdamW/pg1‚7.727303348299275e-06 & lr-AdamW/pg2‚7.727303348299275e-06ðM‚8Í«%Ç % _timestamp‚1.7884149854100955e+09 _runtime‚7595.822254527 _step‚832 & lr-AdamW/pg1‚7.727303348299275e-06 & lr-AdamW/pg2‚7.727303348299275e-06 trainer/global_step‚10149À›[Rß2¬%- _wandbruntime‚7595  _runtime‚7595=þèõí'­%çO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚833 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 135759203 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚300623582047232 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 135758672 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133896465 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14318644130816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚317924967082496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _runtime‚ 7595.82321663 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 339151318 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚286304750913536 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14277540210772 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚303357834559488 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 205254855 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14318813130752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚303357810619392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14566917744128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚286633100932180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14318813130752 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300623582047232 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚300623395044352 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300623395044352 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚317924728363520 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 % _timestamp‚1.7884149854108186e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚286304768916480 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133896463 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040šÍòò'®%ìO \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14318813130752 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300623582047232 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚300623395044352 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300623395044352 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚317924728363520 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 % _timestamp‚1.7884149854108186e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚286304768916480 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133896463 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚833 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 135759203 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚300623582047232 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 135758672 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133896465 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14318644130816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚317924967082496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _runtime‚ 7595.82321663 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 339151318 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚286304750913536 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14277540210772 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚303357834559488 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 205254855 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14318813130752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚303357810619392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14566917744128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚286633100932180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ÁŸ0k£2¯%- _wandbruntime‚7596  _runtime‚7596S¾ÿÔø&°%òM PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 339203830 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚303386723567616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14320037268992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 205294706 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚272381486864384 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚286332038281216 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚317955058039808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚317955292551168 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 135771849 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚286660433279268 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133909122 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 205274642 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14568334472192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 339183764 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884149860684779e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14320037268992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚286660251118296 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14320206282240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚272381503903744 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚498 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚286332038281216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚303386747507712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 339203830 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚286332020278272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57603220 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _step‚834 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14568545043456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135771351 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57603716 _runtime‚7596.481359436ÅHã̱%÷M F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 339203830 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚303386723567616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14320037268992 SEDeviceStatsMonitor.onÿy´¦1#_train_batch_end/allocation.small_pool.allocated‚ 205294706 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚272381486864384 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚286332038281216 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚317955058039808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚317955292551168 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 135771849 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚286660433279268 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133909122 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 205274642 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14568334472192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 339183764 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884149860684779e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14320037268992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚286660251118296 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14320206282240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚272381503903744 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚498 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚286332038281216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚303386747507712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 339203830 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚286332020278272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57603220 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _step‚834 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14568545043456 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135771351 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57603716 _runtime‚7596.481359436 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416Â`˜E2²%- _wandbruntime‚7596  _runtime‚7596¸âaU³%Ï 1loss/SRqqt2/assignment_loss‚1.516502256728184 0loss/FBt3/assignment_loss‚0.2359499089023869 .loss/FRt1/detection_loss‚2.829046903337751 trainer/global_step‚10149 _step‚835 /loss/FRt4/assignment_loss‚2.996030960597244 /loss/FBt2/detection_loss‚2.4506478905677795 /loss/FBt4/detection_loss‚2.0965970121324062 %loss/total_loss‚45.85331860456648 epoch‚8 2loss/SRqqt1/assignment_loss‚2.2421356166402497 2loss/SRqqt3/assignment_loss‚1.2446939375884127 1loss/SRqqt4/detection_loss‚1.8452831548452377 .loss/FBt1/detection_loss‚2.707869757305492 /loss/FRt2/assignment_loss‚2.836474895477295 _runtime‚7596.482668783 0loss/SRqqt1/detection_loss‚2.793127859632174 /loss/FBt3/detection_loss‚0.7654419111728157 /loss/FRt1/assignment_loss‚4.749939380373274 0loss/FRt3/assignment_loss‚1.8485994371500882 0loss/FBt1/assignment_loss‚0.7813976728192012 .loss/FRt2/detection_loss‚2.330614134669304 1loss/SRqqt3/detection_loss‚1.3414301183426156 $ _timestamp‚1.788414986070645e+09 0loss/SRqqt4/assignment_loss‚1.49516457952559 /loss/FRt4/detection_loss‚1.9232124668710373 1loss/SRqqt2/detection_loss‚1.4060394402960656 0loss/FBt2/assignment_loss‚1.2912393130362034 0loss/FBt4/assignment_loss‚0.3510307862825357 /loss/FRt3/detection_loss‚1.7748492102731357k!†Z´%Ô %loss/total_loss‚45.85331860456648 epoch‚8 2loss/SRqqt1/assignment_loss‚2.2421356166402497 2loss/SRqqt3/assignment_loss‚1.2446939375884127 1loss/SRqqt4/detection_loss‚1.8452831548452377 .loss/FBt1/detection_loss‚2.707869757305492 /loss/FRt2/assignment_loss‚2.836474895477295 _runtime‚7596.482668783 0loss/SRqqt1/detection_loss‚2.793127859632174 /loss/FBt3/detection_loss‚0.7654419111728157 /loss/FRt1/assignment_loss‚4.749939380373274 0loss/FRt3/assignment_loss‚1.8485994371500882 0loss/FBt1/assignment_loss‚0.7813976728192012 .loss/FRt2/detection_loss‚2.330614134669304 1loss/SRqqt3/detection_loss‚1.3414301183426156 $ _timestamp‚1.788414986070645e+09 0loss/SRqqt4/assignment_loss‚1.49516457952559 /loss/FRt4/detection_loss‚1.9232124668710373 1loss/SRqqt2/detection_loss‚1.4060394402960656 0loss/FBt2/assignment_loss‚1.2912393130362034 0loss/FBt4/assignment_loss‚0.3510307862825357 /loss/FRt3/detection_loss‚1.7748492102731357 1loss/SRqqt2/assignment_loss‚1.516502256728184 0loss/FBt3/assignment_loss‚0.2359499089023869 .loss/FRt1/detection_loss‚2.829046903337751 trainer/global_step‚10149 _step‚835 /loss/FRt4/assignment_loss‚2.996030960597244 /loss/FBt2/detection_loss‚2.4506478905677795 /loss/FBt4/detection_loss‚2.0965970121324062Ãb£µ%‚(:… ’˜äÔøçž™ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚92.756' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02€353¶%‚(:¨ ’˜äÔ̆ð› disk.dm-1.in‚0 disk.rbd0.out‚0 network.sent‚ 2.4447228e+07) disk./.usagePercent‚73.67880962968448 disk.sda1.out‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚75154.86376953125 disk.drbd1777.in‚0 network.recv‚ 2.806549e+06 disk.sda.out‚ 300.84765625 disk.rbd2.in‚0$ memory_percent‚98.47414493560791# proc.memory.rssMB‚ 2879.48828125 disk.sda.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 249.99609375 disk.rbd1.out‚ 4.048828125! disk.rbd2.out‚174.00341796875" disk.nvme0c0n1.in‚ 711.06640625 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 300.84765625 disk.sda1.in‚0& disk.nvme0n1.out‚75154.86376953125 disk.dm-0.in‚ 711.06640625 disk.dm-1.out‚ 300.84765625 disk.nvme0n1.in‚ 711.06640625# disk.dm-0.out‚75154.86376953125) proc.memory.percent‚17.57500171661377 cpu‚25.38227551373717% disk./.usageGB‚1097.8675079345703 disk.sda2.in‚0 disk.drbd1777.out‚0¢ @‘·%‚(:† ¡˜äԳ湚 gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.398' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03ŒtÖ(¸%‚(: ¡˜äÔÏàÕ› disk.rbd3.out‚0 disk.sda3.out‚ 301.47265625 disk.sda.in‚0# proc.memory.rssMB‚ 2879.48828125 network.sent‚ 2.4556126e+07 disk.rbd3.in‚0 disk.sda2.out‚0 disk.drbd1777.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 301.47265625 disk.rbd1.in‚0% disk.nvme0n1.out‚75253.9638671875" disk.dm-0.out‚75253.9638671875# memory_percent‚98.4771728515625 network.recv‚ 2.813527e+06 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 4.048828125) proc.memory.percent‚17.57500171661377) disk./.usagePercent‚73.67880834964681! disk.rbd2.out‚174.02294921875" disk.nvme0c0n1.in‚ 711.06640625' disk.nvme0c0n1.out‚75253.9638671875 disk.nvme0n1.in‚ 711.06640625 disk.dm-1.out‚ 301.47265625 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda1.out‚0! proc.memory.availableMB‚249.5 cpu‚25.381592221806997 disk.sda2.in‚0$ disk./.usageGB‚1097.867488861084 disk.dm-0.in‚ 711.06640625 proc.cpu.threads‚17rÉAÃ2¹%- _wandbruntime‚7630  _runtime‚7630eâ<Ãʺ%Ä ' lr-AdamW/pg1‚7.3441402610171185e-06 ' lr-AdamW/pg2‚7.3441402610171185e-06 trainer/global_step‚10199 % _timestamp‚1.7884150203557775e+09 _runtime‚7630.767917143 _step‚836R®bÏ»%É trainer/global_step‚10199 % _timestamp‚1.7884150203557775e+09 _runtime‚7630.767917143 _step‚836 ' lr-AdamW/pg1‚7.3441402610171185e-06 ' lr-AdamW/pg2‚7.3441402610171185e-06ÄîPê2¼%- _wandbruntime‚7630  _runtime‚7630—*î'½%èO JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 206267595 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57870147 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14388484453376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚273652719838208 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14346846907776 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚304803481970688 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚287999549706624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚273652702798848 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚287668237153280 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 340776970 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚302056721606656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚837 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57870672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 trainer/global_step‚10199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14388484453376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚287999731867388 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚319441552497152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134529413 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚302056534603776 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚287668219150336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14637866233344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 340797010 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206247557 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 206267595 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚302056534603776 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚287668237153280 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 Ä%%¾%íO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚273652702798848 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚287668237153280 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 340776970 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚302056721606656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚837 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57870672 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 trainer/global_step‚10199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14388484453376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚287999731867388 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚319441552497152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134529413 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚302056534603776 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚287668219150336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14637866233344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 340797010 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206247557 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 206267595 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚302056534603776 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚287668237153280 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 206267595 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57870147 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14388484453376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚273652719838208 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14346846907776 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚304803481970688 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚287999549706624 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0ÅJ,D2¿%- _wandbruntime‚7631  _runtime‚7631©„Ý“÷&À%ñM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 340829482 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 340829482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136413292 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚319471887336960 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚838 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14639288138240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14348401193932 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚304832370978816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚302085384122880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57876309 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚288026882053504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚287695488515072 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 340809416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚288027064214476 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚273678663020544 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14389877604864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134542072 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14389877604864 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14389708591616 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚302085384122880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 206267344 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 $ _timestamp‚1.788415021011152e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78536983 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _runtime‚7631.423990413 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228219904 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134542074 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚287695488515072 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136412767 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚304832394918912 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚287695506518016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78536981 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57875786 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚319471659117056 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 206287408 trainer/global_step‚10199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚273678645981184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14348236072320ûcl‰ü&Á%öM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚838 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14639288138240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14348401193932 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚304832370978816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚302085384122880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57876309 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚288026882053504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚287695488515072 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 340809416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚288027064214476 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚273678663020544 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14389877604864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134542072 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14389877604864 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14389708591616 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚302085384122880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 206267344 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 $ _timestamp‚1.788415021011152e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78536983 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _runtime‚7631.423990413 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228219904 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134542074 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚287695488515072 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136412767 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚304832394918912 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚287695506518016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78536981 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57875786 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚319471659117056 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 206287408 trainer/global_step‚10199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚273678645981184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14348236072320 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 340829482 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 340829482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136413292 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚319471887336960 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Æ9Á‚G2Â%- _wandbruntime‚7631  _runtime‚7631ǯ8ØXÃ%Ò 2loss/SRqqt4/assignment_loss‚1.5780614271759987 /loss/FRt1/detection_loss‚2.8461271459991866 .loss/FBt3/detection_loss‚1.012390549285635 /loss/FBt4/detection_loss‚2.6038456857204437 &loss/total_loss‚46.008626599691766 /loss/FRt1/assignment_loss‚4.765932508417078 1loss/SRqqt2/detection_loss‚1.6216242515123807 /loss/FBt2/detection_loss‚2.2165443130901883 /loss/FRt4/detection_loss‚2.1562267492214837 .loss/FRt2/detection_loss‚2.287106537156635 /loss/FRt2/assignment_loss‚2.773487389087677 0loss/SRqqt1/assignment_loss‚2.28261764219829 /loss/FRt3/detection_loss‚1.8097921143852027 0loss/FBt3/assignment_loss‚0.2636640722552807 1loss/SRqqt3/detection_loss‚1.2479532113564866 0loss/SRqqt4/detection_loss‚1.794273471290415 epoch‚8 trainer/global_step‚10199 _step‚839 0loss/FRt3/assignment_loss‚2.1720752213515486 0loss/SRqqt3/assignment_loss‚1.42597416928038 0loss/FBt1/assignment_loss‚0.6033526379817431 1loss/FBt4/assignment_loss‚0.23673696622095122 _runtime‚7631.425588604 2loss/SRqqt2/assignment_loss‚1.5052830127569345 0loss/FBt2/assignment_loss‚0.5240196821552152 /loss/FBt1/detection_loss‚2.4601528458297253 0loss/FRt4/assignment_loss‚3.1519305780529976 1loss/SRqqt1/detection_loss‚2.6694544179098947 $ _timestamp‚1.788415021013504e+09*B#r]Ä%× /loss/FBt1/detection_loss‚2.4601528458297253 0loss/FRt4/assignment_loss‚3.1519305780529976 1loss/SRqqt1/detection_loss‚2.6694544179098947 $ _timestamp‚1.788415021013504e+09 2loss/SRqqt4/assignment_loss‚1.5780614271759987 /loss/FRt1/detection_loss‚2.8461271459991866 .loss/FBt3/detection_loss‚1.012390549285635 /loss/FBt4/detection_loss‚2.6038456857204437 &loss/total_loss‚46.008626599691766 /loss/FRt1/assignment_loss‚4.765932508417078 1loss/SRqqt2/detection_loss‚1.6216242515123807 /loss/FBt2/detection_loss‚2.2165443130901883 /loss/FRt4/detection_loss‚2.1562267492214837 .loss/FRt2/detection_loss‚2.287106537156635 /loss/FRt2/assignment_loss‚2.773487389087677 0loss/SRqqt1/assignment_loss‚2.28261764219829 /loss/FRt3/detection_loss‚1.8097921143852027 0loss/FBt3/assignment_loss‚0.2636640722552807 1loss/SRqqt3/detection_loss‚1.2479532113564866 0loss/SRqqt4/detection_loss‚1.794273471290415 epoch‚8 trainer/global_step‚10199 _step‚839 0loss/FRt3/assignment_loss‚2.1720752213515486 0loss/SRqqt3/assignment_loss‚1.42597416928038 0loss/FBt1/assignment_loss‚0.6033526379817431 1loss/FBt4/assignment_loss‚0.23673696622095122 _runtime‚7631.425588604 2loss/SRqqt2/assignment_loss‚1.5052830127569345 0loss/FBt2/assignment_loss‚0.5240196821552152ÇŠŠ‘Å%‚(:† °˜äÔ½©¨™ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.163' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MVÀF.Æ%‚(:£ °˜äÔÇ¿´œ network.sent‚ 2.4561454e+07 disk.drbd1777.out‚0% disk.nvme0n1.out‚75353.0810546875 disk.rbd2.in‚0 disk.sda.in‚0" disk.dm-0.out‚75353.0810546875 disk.sda.out‚ 301.93359375 network.recv‚ 2.817501e+06" disk.nvme0c0n1.in‚ 711.06640625 proc.cpu.threads‚17( disk./.usagePercent‚73.6788119337523' disk.nvme0c0n1.out‚75353.0810546875 disk.sda3.in‚0 disk.sda3.out‚ 301.93359375 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd3.out‚0% disk./.usageGB‚1097.8675422668457 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.dm-1.in‚0! disk.rbd2.out‚175.65966796875 disk.rbd1.out‚ 4.048828125 disk.nvme0n1.in‚ 711.06640625 disk.rbd0.out‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 249.7265625# proc.memory.rssMB‚ 2879.48828125 cpu‚25.38093779286151 disk.dm-0.in‚ 711.06640625 disk.dm-1.out‚ 301.93359375 disk.rbd0.in‚0$ memory_percent‚98.47579002380371) proc.memory.percent‚17.57500171661377ÈVÇ%‚(:… ¿˜äÔ˜­™ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.957' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7914 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z|&È%‚(:› ¿˜äÔÕÓ•œ% disk./.usageGB‚1097.8675498962402 disk.sda3.out‚ 302.796875 disk.sda1.out‚0 disk.dm-1.out‚ 302.796875 disk.sda2.out‚0 disk.sda.in‚0$ disk.nvme0n1.out‚75452.455078125 disk.rbd3.in‚0& disk.nvme0c0n1.out‚75452.455078125 disk.rbd1.out‚ 4.048828125 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 711.06640625 proc.cpu.threads‚17 disk.sda1.in‚0! disk.dm-0.out‚75452.455078125 disk.rbd0.in‚0 disk.rbd0.out‚0) proc.memory.percent‚17.57500171661377 disk.sda.out‚ 302.796875! disk.rbd2.out‚175.67529296875# proc.memory.rssMB‚ 2879.48828125 cpu‚25.38029263888213 network.recv‚ 2.824052e+06 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 711.06640625 disk.sda2.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 711.06640625 disk.dm-1.in‚0) disk./.usagePercent‚73.67881244576736 disk.rbd3.out‚0$ memory_percent‚98.47452640533447( proc.memory.availableMB‚ 249.93359375 network.sent‚ 2.466977e+07LË}É%‚(:… ΘäÔàô³™ gpu.0.gpu‚8.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.899' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3798 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YÍ(Ê%‚(: ΘäÔüÆŸœ disk.dm-1.out‚ 303.7109375 disk.rbd2.in‚0 disk.sda.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 711.06640625 network.sent‚ 2.4675997e+07 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 249.93359375 cpu‚25.37964778936905 proc.cpu.threads‚17 disk.rbd1.out‚ 4.048828125( disk./.usagePercent‚73.6787379475749$ disk./.usageGB‚1097.866439819336! disk.rbd2.out‚175.75341796875 disk.rbd3.out‚0 disk.sda3.out‚ 303.7109375! disk.dm-0.out‚75454.439453125# proc.memory.rssMB‚ 2879.48828125 network.recv‚ 2.829125e+06 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 711.06640625 disk.sda2.in‚0) proc.memory.percent‚17.57500171661377& disk.nvme0c0n1.out‚75454.439453125 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚75454.439453125$ memory_percent‚98.47452640533447 disk.sda.out‚ 303.7109375 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 711.06640625dÃIæ2Ë%- _wandbruntime‚7666  _runtime‚7666­åøÈÌ% & lr-AdamW/pg1‚6.969974413761771e-06 & lr-AdamW/pg2‚6.969974413761771e-06 trainer/global_step‚10249 % _timestamp‚1.7884150557791705e+09 _runtime‚7666.191328984 _step‚840W[ÚÄÍÍ%Ç trainer/global_step‚10249 % _timestamp‚1.7884150557791705e+09 _runtime‚7666.191328984 _step‚840 & lr-AdamW/pg1‚6.969974413761771e-06 & lr-AdamW/pg2‚6.969974413761771e-06ÈøZâ92Î%- _wandbruntime‚7666  _runtime‚7666ÆÄ8‹‚Ï%çO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚306249129381888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137040756 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚289031687387136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚515 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 207240259 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14416318726184 N@DeviceStatsMonitor.on_train_batch_start/inactive_spl±Àúk it.all.freed‚ 137040239 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 342402622 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78898456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58142300 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14457986776064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14416483847588 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 trainer/global_step‚10249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135162363 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚303489674163200 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58141785 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14457986776064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚289031687387136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚306249105441792 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 207260297 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚274949878955008 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚320958177752064 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14708833591296 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚274949861915648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚517 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78898454 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚303489861166080 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚303489861166080 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040/Ùò'Ð%ìO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 342402622 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78898456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58142300 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14457986776064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14416483847588 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 trainer/global_step‚10249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135162363 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚303489674163200 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58141785 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14457986776064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚289031687387136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚306249105441792 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 207260297 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚274949878955008 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚320958177752064 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14708833591296 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚274949861915648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚517 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78898454 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚303489861166080 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚303489861166080 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚306249129381888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137040756 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚289031687387136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚515 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 207240259 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14416318726184 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137040239 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416É`*Ð2Ñ%- _wandbruntime‚7666  _runtime‚7666bŠø&Ò%òM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 342435068 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135175024 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14459379914240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78905833 trainer/global_step‚10249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14417707890728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78905831 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 _step‚842 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚303518523682304 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚289058974754816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137052999 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342455134 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚289058974754816 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14710463941120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚303518336666112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚303518523682304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14710251272704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14459548927488 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135175022 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137053531 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 207280110 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚320988506271232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚306278018390016 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚289393512988712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58147698 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚7666.848008314 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 342435068 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚289393695149684 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚274975805097984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14459548927488 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14459379914240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135175022 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58147168 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚289058956751872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 342455134 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066oîÐVÓ%÷M _runtime‚7666.848008314 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 342435068 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚289393695149684 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚274975805097984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14459548927488 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14459379914240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135175022 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58147168 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚289058956751872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 342455134 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 342435068 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135175024 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14459379914240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78905833 trainer/global_step‚10249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14417707890728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78905831 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 _step‚842 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚303518523682304 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚289058974754816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137052999 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342455134 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚289058974754816 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14710463941120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚303518336666112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚303518523682304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14710251272704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14459548927488 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135175022 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137053531 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 207280110 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚320988506271232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚306278018390016 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚289393512988712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58147698 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0Êü³{Å2Ô%- _wandbruntime‚7666  _runtime‚7666¾dœvZÕ%Ô /loss/FRt3/assignment_loss‚1.823629074153446 /loss/FBt4/detection_loss‚1.9143557647864025 epoch‚8 % _timestamp‚1.7884150564377081e+09 2loss/SRqqt4/assignment_loss‚1.2630827073366022 1loss/FBt1/assignment_loss‚0.25986371932945723 1loss/SRqqt3/detection_loss‚1.3998759295791388 .loss/FBt1/detection_loss‚2.818325766495296 2loss/SRqqt1/assignment_loss‚2.3678355133533477 2loss/SRqqt3/assignment_loss‚1.4238527202202629 0loss/FBt2/assignment_loss‚0.8927281275355957 /loss/FRt1/detection_loss‚2.9989968027387346 1loss/SRqqt2/detection_loss‚1.4743626845940467 /loss/FBt3/detection_loss‚0.8047732235280998 0loss/SRqqt2/assignment_loss‚1.49295096892503 0loss/FBt4/assignment_loss‚0.3721351359559624 .loss/FRt2/detection_loss‚2.437940262935378 /loss/FRt1/assignment_loss‚5.069691019398825 _runtime‚7666.849781666 /loss/FRt4/assignment_loss‚2.739661357751707 .loss/FRt4/detection_loss‚2.286276442248647 trainer/global_step‚10249 /loss/FRt3/detection_loss‚2.0128518974497203 1loss/SRqqt1/detection_loss‚2.8754596376419066 &loss/total_loss‚46.860754879840314 0loss/FRt2/assignment_loss‚3.0739995037967507 1loss/FBt3/assignment_loss‚0.32097543893681874 0loss/SRqqt4/detection_loss‚1.797471963155728 /loss/FBt2/detection_loss‚2.9396592179934182 _step‚843úèÑó_Ö%Ù /loss/FBt3/detection_loss‚0.8047732235280998 0loss/SRqqt2/assignment_loss‚1.49295096892503 0loss/FBt4/assignment_loss‚0.3721351359559624 .loss/FRt2/detection_loss‚2.437940262935378 /loss/FRt1/assignment_loss‚5.069691019398825 _runtime‚7666.849781666 /loss/FRt4/assignment_loss‚2.739661357751707 .loss/FRt4/detection_loss‚2.286276442248647 trainer/global_step‚10249 /loss/FRt3/detection_loss‚2.0128518974497203 1loss/SRqqt1/detection_loss‚2.8754596376419066 &loss/total_loss‚46.860754879840314 0loss/FRt2/assignment_loss‚3.0739995037967507 1loss/FBt3/assignment_loss‚0.32097543893681874 0loss/SRqqt4/detection_loss‚1.797471963155728 /loss/FBt2/detection_loss‚2.9396592179934182 _step‚843 /loss/FRt3/assignment_loss‚1.823629074153446 /loss/FBt4/detection_loss‚1.9143557647864025 epoch‚8 % _timestamp‚1.7884150564377081e+09 2loss/SRqqt4/assignment_loss‚1.2630827073366022 1loss/FBt1/assignment_loss‚0.25986371932945723 1loss/SRqqt3/detection_loss‚1.3998759295791388 .loss/FBt1/detection_loss‚2.818325766495296 2loss/SRqqt1/assignment_loss‚2.3678355133533477 2loss/SRqqt3/assignment_loss‚1.4238527202202629 0loss/FBt2/assignment_loss‚0.8927281275355957 /loss/FRt1/detection_loss‚2.9989968027387346 1loss/SRqqt2/detection_loss‚1.4743626845940467ËÜÀ(§×%‚(:… ݘäÔ¤œ³™ gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.035' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.807 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍQü£4Ø%‚(:© ݘäÔ¹¶¸ network.recv‚ 2.833511e+06 disk.rbd1.in‚0& disk.nvme0n1.out‚75556.56689453125 disk.rbd3.out‚0( disk.nvme0c0n1.out‚75556.56689453125 disk.sda1.out‚0 disk.drbd1777.in‚0 cpu‚25.378965382368353 disk.rbd1.out‚ 4.048828125 disk.rbd0.out‚0! disk.rbd2.out‚175.76904296875 disk.sda2.out‚0 disk.sda3.out‚ 304.28515625% disk./.usageGB‚1097.8671493530273 disk.dm-1.out‚ 304.28515625 disk.sda2.in‚0# proc.memory.rssMB‚ 2879.48828125 disk.dm-0.in‚ 713.92578125 disk.rbd3.in‚0 disk.dm-1.in‚0 disk.sda3.in‚0 network.sent‚ 2.4682073e+07) disk./.usagePercent‚73.67878556497627# disk.dm-0.out‚75556.56689453125 disk.sda.in‚0 disk.drbd1777.out‚0$ memory_percent‚98.47638607025146 disk.nvme0n1.in‚ 713.92578125 disk.sda.out‚ 304.28515625 disk.sda1.in‚0) proc.memory.percent‚17.57500171661377 disk.rbd0.in‚0 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 713.92578125( proc.memory.availableMB‚ 249.62890625 proc.cpu.threads‚17~•NœÙ%‚(:‘ ì˜äÔ‚á•™ gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚112.031' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.406200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïUc%Ú%‚(:” ì˜äÔ¡Õœ) disk./.usagePercent‚73.67878351691598% disk./.usageGB‚1097.8671188354492 disk.rbd0.in‚0 disk.rbd2.out‚ 175.98046875 disk.rbd3.out‚0 disk.sda1.out‚0$ disk.nvme0n1.out‚75655.677734375 disk.dm-0.in‚ 714.0703125! disk.dm-0.out‚75655.677734375 disk.rbd0.out‚0 disk.nvme0n1.in‚ 714.0703125# proc.memory.rssMB‚ 2879.48828125 disk.rbd2.in‚0& disk.nvme0c0n1.out‚75655.677734375 disk.sda.out‚ 304.78125 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 714.0703125 disk.dm-1.out‚ 304.78125) proc.memory.percent‚17.57500171661377 cpu‚25.378367533068417 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 2.840171e+06 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.sda3.out‚ 304.78125 disk.rbd1.out‚ 4.048828125$ memory_percent‚98.47896099090576( proc.memory.availableMB‚ 249.20703125 network.sent‚ 2.4790457e+07 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚0ž€[·2Û%- _wandbruntime‚7701  _runtime‚7701@Ò‘ÛÅÜ%¿ % lr-AdamW/pg2‚6.60488378213499e-06 trainer/global_step‚10299 $ _timestamp‚1.788415091058952e+09 _runtime‚7701.471250192 _step‚844 % lr-AdamW/pg1‚6.60488378213499e-06¨Ô)ÊÝ%Ä % lr-AdamW/pg1‚6.60488378213499e-06 % lr-AdamW/pg2‚6.60488378213499e-06 trainer/global_step‚10299 $ _timestamp‚1.788415091058952e+09 _runtime‚7701.471250192 _step‚844Ìðh2Þ%- _wandbruntime‚7701  _runtime‚7701HMJãî'ß%èO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79267306 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14779777114112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚322474772880384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚290732993737804 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 344028274 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135795313 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚290395173626880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚555 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137681837 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 208232961 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304922813722624 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚553 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚307694776793088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚322474529967104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14485955665996 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14527827098624 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚276247021032448 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14485790544592 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 344048314 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚307694752852992 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14779996087296 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135795313 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚290395173626880 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 trainer/global_step‚10299 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14527827098624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58415086 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135795315 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135795315 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 % _timestamp‚1.7884150910597541e+09 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚290395173626880 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 trainer/global_step‚10299 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14527827098624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58415086 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135795315 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135795315 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 % _timestamp‚1.7884150910597541e+09 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚322474772880384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚290732993737804 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 344028274 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135795313 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚290395173626880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚555 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137681837 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 208232961 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304922813722624 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚553 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚307694776793088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚322474529967104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14485955665996 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14527827098624 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚276247021032448 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14485790544592 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 344048314 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚307694752852992ÍQËl—2á%- _wandbruntime‚7702  _runtime‚7702Ãb´=yâ%òM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14529220250112 C9DeviceStatsMonitor.‰áw^&on_train_batch_end/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _step‚846 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322504861894144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 344080786 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚322505096405504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135807972 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚290422442991616 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14487344830748 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚307723665801216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 208272812 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14487179709136 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135807974 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 208252748 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 344080786 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 344060720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14529220250112 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14781406664192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _runtime‚7702.127684075 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14529051236864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚307723689741312 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14781196092928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14529051236864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚290422442991616 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 208272812 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚304951476225536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135807972 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 trainer/global_step‚10299 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137695166 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137694627 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚290422424988672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚290422424988672Ý×Pãý&ã%÷M WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚290422424988672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚290422424988672 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14529220250112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _step‚846 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322504861894144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 344080786 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚322505096405504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135807972 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚290422442991616 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14487344830748 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚307723665801216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 208272812 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14487179709136 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135807974 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 208252748 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 344080786 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 344060720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14529220250112 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14781406664192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _runtime‚7702.127684075 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14529051236864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚307723689741312 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14781196092928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14529051236864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚290422442991616 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 208272812 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚304951476225536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135807972 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 trainer/global_step‚10299 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137695166 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137694627 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160ÎÍRÇH2ä%- _wandbruntime‚7702  _runtime‚7702¡¬ÜTå%Î .loss/FBt1/detection_loss‚2.555950054996892 $ _timestamp‚1.788415091717132e+09 0loss/FRt2/assignment_loss‚3.4401879920515905 0loss/FRt4/assignment_loss‚3.3336372961077774 /loss/FRt3/detection_loss‚1.7505395747721195 0loss/SRqqt1/detection_loss‚2.752777871158388 .loss/FBt4/detection_loss‚2.577753223478794 _runtime‚7702.129200913 /loss/FRt1/assignment_loss‚4.926854035433601 2loss/SRqqt2/assignment_loss‚1.7777595236257184 /loss/FBt2/detection_loss‚2.8540512160821394 %loss/total_loss‚47.90346871264781 /loss/FRt2/detection_loss‚2.3228264083695964 .loss/FRt4/detection_loss‚2.060750361597329 _step‚847 .loss/FRt1/detection_loss‚3.072623387855642 0loss/FBt1/assignment_loss‚0.5188361417998462 0loss/SRqqt3/detection_loss‚1.473512300218527 /loss/FBt3/detection_loss‚0.7239657736766865 2loss/SRqqt3/assignment_loss‚1.4984373592126827 2loss/SRqqt4/assignment_loss‚1.4456925475496356 0loss/FBt4/assignment_loss‚0.6142146628699265 trainer/global_step‚10299 /loss/FBt2/assignment_loss‚0.512773089801075 0loss/FBt3/assignment_loss‚0.2808150867944572 1loss/SRqqt2/detection_loss‚1.4304601995972916 epoch‚8 /loss/FRt3/assignment_loss‚2.135472639747288 0loss/SRqqt1/assignment_loss‚2.27658008646082 1loss/SRqqt4/detection_loss‚1.5669978793899892Ê6^PYæ%Ó $ _timestamp‚1.788415091717132e+09 0loss/FRt2/assignment_loss‚3.4401879920515905 0loss/FRt4/assignment_loss‚3.3336372961077774 /loss/FRt3/detection_loss‚1.7505395747721195 0loss/SRqqt1/detection_loss‚2.752777871158388 .loss/FBt4/detection_loss‚2.577753223478794 _runtime‚7702.129200913 /loss/FRt1/assignment_loss‚4.926854035433601 2loss/SRqqt2/assignment_loss‚1.7777595236257184 /loss/FBt2/detection_loss‚2.8540512160821394 %loss/total_loss‚47.90346871264781 /loss/FRt2/detection_loss‚2.3228264083695964 .loss/FRt4/detection_loss‚2.060750361597329 _step‚847 .loss/FRt1/detection_loss‚3.072623387855642 0loss/FBt1/assignment_loss‚0.5188361417998462 0loss/SRqqt3/detection_loss‚1.473512300218527 /loss/FBt3/detection_loss‚0.7239657736766865 2loss/SRqqt3/assignment_loss‚1.4984373592126827 2loss/SRqqt4/assignment_loss‚1.4456925475496356 0loss/FBt4/assignment_loss‚0.6142146628699265 trainer/global_step‚10299 /loss/FBt2/assignment_loss‚0.512773089801075 0loss/FBt3/assignment_loss‚0.2808150867944572 1loss/SRqqt2/detection_loss‚1.4304601995972916 epoch‚8 /loss/FRt3/assignment_loss‚2.135472639747288 0loss/SRqqt1/assignment_loss‚2.27658008646082 1loss/SRqqt4/detection_loss‚1.5669978793899892 .loss/FBt1/detection_loss‚2.555950054996892ÏG!¼›Žç%‚(:ƒ û˜äÔ—ɬ™ gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚99.07' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.814 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥úÉ'è%‚(:œ û˜äÔ¾¸Óœ disk.sda1.out‚0 disk.drbd1777.in‚0 network.recv‚ 2.843825e+06 disk.nvme0c0n1.in‚ 714.828125 disk.rbd1.in‚0& disk.nvme0n1.out‚75756.42138671875$ memory_percent‚98.47760200500488 proc.cpu.threads‚17 disk.rbd2.out‚ 175.98046875 disk.rbd3.out‚0 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚75756.42138671875 disk.sda.in‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0# disk.dm-0.out‚75756.42138671875 network.sent‚ 2.4796057e+07% disk./.usageGB‚1097.8681755065918 disk.rbd1.out‚ 4.048828125 disk.nvme0n1.in‚ 714.828125' proc.memory.availableMB‚ 249.4296875 cpu‚25.377727124293454 disk.sda3.in‚0) proc.memory.percent‚17.57500171661377 disk.dm-1.in‚0 disk.sda.out‚ 305.4453125 disk.dm-0.in‚ 714.828125# proc.memory.rssMB‚ 2879.48828125) disk./.usagePercent‚73.67885443100299 disk.dm-1.out‚ 305.4453125 disk.sda2.in‚0 disk.sda3.out‚ 305.4453125OleØœé%‚(:‘ Š™äÔ¾™É™ gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.933' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.186600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿¦^$ê%‚(:™ Š™äÔ˜€Öœ network.recv‚ 2.851509e+06 disk.sda2.out‚0 disk.rbd1.out‚ 4.048828125$ memory_percent‚98.47900867462158# proc.memory.rssMB‚ 2879.48828125) proc.memory.percent‚17.57500171661377% disk./.usageGB‚1097.8682174682617 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd2.out‚ 177.6953125 disk.nvme0c0n1.in‚ 714.828125 cpu‚25.377060399535427 disk.sda.out‚ 305.890625 disk.dm-1.in‚0 disk.dm-1.out‚ 305.890625 disk.sda.in‚0 disk.rbd2.in‚0( disk.nvme0c0n1.out‚75855.91455078125 proc.cpu.threads‚17) disk./.usagePercent‚73.67885724708586 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 714.828125 disk.dm-0.in‚ 714.828125# disk.dm-0.out‚75855.91455078125 disk.rbd0.in‚0 network.sent‚ 2.4905046e+07 disk.sda1.in‚0& disk.nvme0n1.out‚75855.91455078125 disk.rbd0.out‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 249.19921875 disk.sda3.out‚ 305.890625Hùê2ë%- _wandbruntime‚7736  _runtime‚7736dxþ¸Êì%Ä ' lr-AdamW/pg2‚6.2489444504772045e-06 trainer/global_step‚10349 % _timestamp‚1.7884151263536696e+09 _runtime‚7736.765824079 _step‚848 ' lr-AdamW/pg1‚6.2489444504772045e-06xø*Ïí%É ' lr-AdamW/pg1‚6.2489444504772045e-06 ' lr-AdamW/pg2‚6.2489444504772045e-06 trainer/global_step‚10349 % _timestamp‚1.7884151263536696e+09 _runtime‚7736.765824079 _step‚848ÐÔ™R52î%- _wandbruntime‚7736  _runtime‚7736#£$7Ñï%èO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14850917101568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136428263 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚291758641863680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚277544180149248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚291758641863680 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚306355953282048 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚277544197188608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 trainer/global_step‚10349 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚323991341305856 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 209245701 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136428263 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138323168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚292099624673012 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚291758623860736 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚306356140284928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58687014 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 % _timestamp‚1.7884151263544264e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/acä >tive.small_pool.freed‚ 209225663 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 345673966 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79636156 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚292099442512248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚309140400264192 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14597498421248 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14850917101568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136428263 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚291758641863680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚277544180149248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚291758641863680 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚306355953282048 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚277544197188608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 trainer/global_step‚10349 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚323991341305856 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 209245701 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136428263 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138323168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚292099624673012 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚291758623860736 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚306356140284928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58687014 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 % _timestamp‚1.7884151263544264e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 209225663 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 345673966 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79636156 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚292099442512248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚309140400264192 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14597498421248 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 345706438 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79643533 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚306384615784960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚291785893225472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 % _timestamp‚1.7884151270151086e+09 _runtime‚7737.427956605 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚291785893225472 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14598891572736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14598722559488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚291785911228416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14852335605248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 345686372 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138335819 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚309169337152512 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136440924 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136440922 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 209265514 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 _step‚850 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14556816649156 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58692288 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚292126957020100 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14598891572736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79643531 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14556651527544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚306384802801152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚291785911228416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 209245450 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14598722559488 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 345706438 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 209245450 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14852131325440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136440924 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚277570140370944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚277570123331584 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 138336350 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 trainer/global_step‚10349 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚306384802801152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248Ýyn£"ó%÷M E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚291785911228416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14852335605248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 345686372 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138335819 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚309169337152512 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136440924 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136440922 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 209265514 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 _step‚850 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14556816649156 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58692288 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚292126957020100 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14598891572736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79643531 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14556651527544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚306384802801152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚291785911228416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 209245450 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14598722559488 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 345706438 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 209245450 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14852131325440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136440924 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚277570140370944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚277570123331584 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 138336350 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 trainer/global_step‚10349 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚306384802801152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 209265514 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58692817 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚309169313212416 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 345706438 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79643533 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚306384615784960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚291785893225472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 % _timestamp‚1.7884151270151086e+09 _4jZruntime‚7737.427956605 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚291785893225472 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14598891572736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14598722559488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416Ò{«Û2ô%- _wandbruntime‚7737  _runtime‚7737mEd[õ%Õ .loss/FRt3/detection_loss‚1.789009657377998 /loss/FBt3/detection_loss‚0.7231723146622627 _step‚851 2loss/SRqqt1/assignment_loss‚2.0197162236417494 0loss/FRt2/assignment_loss‚3.1348651134967804 %loss/total_loss‚47.90328428416595 .loss/FRt3/assignment_loss‚2.38135731096069 /loss/FBt1/assignment_loss‚0.622642945123847 /loss/FRt2/detection_loss‚2.2642603150010108 /loss/FBt1/detection_loss‚2.9625506455248054 % _timestamp‚1.7884151270171256e+09 /loss/FRt1/assignment_loss‚5.035814676965986 1loss/SRqqt3/assignment_loss‚1.273089597533856 2loss/SRqqt4/assignment_loss‚1.6230374764178008 0loss/FBt2/assignment_loss‚0.2729334661043039 1loss/FBt4/assignment_loss‚0.44989834191049094 .loss/FRt4/detection_loss‚2.102955442528392 1loss/SRqqt1/detection_loss‚2.7140040416871347 0loss/FRt4/assignment_loss‚3.3056166532427764 1loss/FBt3/assignment_loss‚0.17827694709256106 1loss/SRqqt2/detection_loss‚1.6971418999601156 1loss/SRqqt4/detection_loss‚1.5944399516733865 /loss/FBt2/detection_loss‚2.8964686195055642 trainer/global_step‚10349 1loss/SRqqt3/detection_loss‚1.3963543215234364 /loss/FBt4/detection_loss‚3.1299597431312907 epoch‚8 2loss/SRqqt2/assignment_loss‚1.4382563728722744 /loss/FRt1/detection_loss‚2.8974622062274387 _runtime‚7737.429185139x?À`ö%Ú _runtime‚7737.429185139 .loss/FRt3/detection_loss‚1.789009657377998 /loss/FBt3/detection_loss‚0.7231723146622627 _step‚851 2loss/SRqqt1/assignment_loss‚2.0197162236417494 0loss/FRt2/assignment_loss‚3.1348651134967804 %loss/total_loss‚47.90328428416595 .loss/FRt3/assignment_loss‚2.38135731096069 /loss/FBt1/assignment_loss‚0.622642945123847 /loss/FRt2/detection_loss‚2.2642603150010108 /loss/FBt1/detection_loss‚2.9625506455248054 % _timestamp‚1.7884151270171256e+09 /loss/FRt1/assignment_loss‚5.035814676965986 1loss/SRqqt3/assignment_loss‚1.273089597533856 2loss/SRqqt4/assignment_loss‚1.6230374764178008 0loss/FBt2/assignment_loss‚0.2729334661043039 1loss/FBt4/assignment_loss‚0.44989834191049094 .loss/FRt4/detection_loss‚2.102955442528392 1loss/SRqqt1/detection_loss‚2.7140040416871347 0loss/FRt4/assignment_loss‚3.3056166532427764 1loss/FBt3/assignment_loss‚0.17827694709256106 1loss/SRqqt2/detection_loss‚1.6971418999601156 1loss/SRqqt4/detection_loss‚1.5944399516733865 /loss/FBt2/detection_loss‚2.8964686195055642 trainer/global_step‚10349 1loss/SRqqt3/detection_loss‚1.3963543215234364 /loss/FBt4/detection_loss‚3.1299597431312907 epoch‚8 2loss/SRqqt2/assignment_loss‚1.4382563728722744 /loss/FRt1/detection_loss‚2.8974622062274387Óp‡â‘÷%‚(:† ™™äԠ츙 gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8658 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»þn"ø%‚(:— ™™äÔÊË–œ network.recv‚ 2.855955e+06 disk.rbd1.in‚0) proc.memory.percent‚17.57500171661377 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 715.546875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd1.out‚ 4.048828125% disk.nvme0n1.out‚75857.2568359375 disk.dm-1.out‚ 306.4765625 disk.rbd2.out‚ 177.703125 disk.sda.out‚ 306.4765625 disk.drbd1777.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 cpu‚25.37639971670161% disk./.usageGB‚1097.8683586120605 disk.nvme0n1.in‚ 715.546875" disk.dm-0.out‚75857.2568359375 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 715.546875 network.sent‚ 2.4911156e+07 disk.sda3.out‚ 306.4765625 disk.sda1.in‚0 disk.drbd1777.out‚0$ memory_percent‚98.47919940948486( proc.memory.availableMB‚ 249.16796875) disk./.usagePercent‚73.67886671936462' disk.nvme0c0n1.out‚75857.2568359375 disk.sda.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17# proc.memory.rssMB‚ 2879.48828125Ž{œù%‚(:‘ ¨™äÔûí¤™ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.799' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.159800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃßÛäú%‚(:“ ¨™äÔ’ëÊœ# proc.memory.rssMB‚ 2879.48828125 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 306.890625 disk.sda1.in‚0 cpu‚25.37576764271576 network.sent‚ 2.5019316e+07 network.recv‚ 2.862522e+06 disk.rbd0.out‚0 disk.sda.out‚ 306.890625 disk.sda1.out‚0 disk.nvme0n1.in‚ 715.546875$ memory_percent‚98.47850799560547 disk.dm-1.out‚ 306.890625 disk.rbd0.in‚0 disk.dm-0.in‚ 715.546875) proc.memory.percent‚17.57500171661377) disk./.usagePercent‚73.67886671936462' disk.nvme0c0n1.out‚75956.3857421875 disk.rbd1.out‚ 4.048828125 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.nvme0c0n1.in‚ 715.546875 disk.sda2.out‚0 disk.sda.in‚0" disk.dm-0.out‚75956.3857421875% disk./.usageGB‚1097.8683586120605 disk.dm-1.in‚0 disk.rbd2.out‚ 177.80859375 disk.rbd1.in‚0% disk.nvme0n1.out‚75956.3857421875% proc.memory.availableMB‚ 249.28125×Úû|‘û%‚(:† ·™äÔó¸™ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.419' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4838 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L4’Îü%‚(:’ ·™äÔ­¡Í disk.dm-1.out‚ 307.5703125 disk.drbd1777.out‚0% disk./.usageGB‚1097.8684387207031 disk.rbd0.in‚0 disk.sda.out‚ 307.5703125 disk.rbd0.out‚0# disk.nvme0n1.out‚76055.86328125 disk.rbd3.in‚0 disk.rbd3.out‚0 cpu‚25.375133833233516 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd1.out‚ 4.048828125 disk.rbd2.in‚0 disk.rbd2.out‚ 177.80859375) disk./.usagePercent‚73.67887209552283 disk.nvme0n1.in‚ 715.546875 disk.dm-0.out‚76055.86328125 disk.nvme0c0n1.in‚ 715.546875% disk.nvme0c0n1.out‚76055.86328125 disk.sda2.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 307.5703125 disk.sda.in‚0 disk.sda1.in‚0& proc.memory.availableMB‚ 249.359375# proc.memory.rssMB‚ 2879.48828125 network.sent‚ 2.5024941e+07 network.recv‚ 2.866181e+06 disk.dm-0.in‚ 715.546875 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0$ memory_percent‚98.47803115844727) proc.memory.percent‚17.57500171661377RÀÁî2ý%- _wandbruntime‚7771  _runtime‚7771 øEÅþ%¿ % lr-AdamW/pg1‚5.90223059601161e-06 % lr-AdamW/pg2‚5.90223059601161e-06 trainer/global_step‚10399 $ _timestamp‚1.788415161558179e+09 _runtime‚7771.970342744 _step‚852¤SpÊÿ%Ä _runtime‚7771.970342744 _step‚852 % lr-AdamW/pg1‚5.90223059601161e-06 % lr-AdamW/pg2‚5.90223059601161e-06 trainer/global_step‚10399 $ _timestamp‚1.788415161558179e+09ÔE‚«2€&- _wandbruntime‚7771  _runtime‚7771°„'Îî'&èO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚293122110100480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14624734181408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14921876396544 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138963377 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 % _timestamp‚1.7884151615589826e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 _runtime‚7771.971287034 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80005006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14667000743936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58958907 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚310586071615488 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚307789092841472 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137061213 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚325507713487360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 trainer/global_step‚10399 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14667169743872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚146671697438724‚±ó'‚&íO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚293466255608220 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138963377 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 % _timestamp‚1.7884151615589826e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 _runtime‚7771.971287034 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80005006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14667000743936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58958907 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚310586071615488 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚307789092841472 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137061213 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚325507713487360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 trainer/global_step‚10399 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14667169743872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14667169743872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚293122110100480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14624734181408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14921876396544 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚293493587955308 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚293149361462272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14626288467564 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 % _timestamp‚1.7884151622289336e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚293149379465216 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14668393882112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 UHDeviceStatsMoFÓ+Þnitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80012381 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚307817755344384 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14923294529024 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚278867282448384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚278867299487744 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 347312024 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14668562895360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137073872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14668562895360 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 138976761 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _runtime‚7772.641630716 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137073874 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137073874 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚307817942360576 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 210258216 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚310614960623616 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 210238152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14923083957760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚854 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚293149361462272 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚10399 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 210238152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347332090 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚325538279092736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚310614984563712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58964378 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 347312024 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138976237 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14626123345952 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚307817942360576 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347332090=Ž¬ý&…&÷M KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347332090 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚325538279092736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚310614984563712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58964378 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 347312024 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138976237 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14626123345952 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚307817942360576 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347332090 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58963856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚293149379465216 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚293493587955308 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚293149361462272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14626288467564 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 % _timestamp‚1.7884151622289336e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚293149379465216 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14668393882112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80012381 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚307817755344384 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14923294529024 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚278867282448384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚278867299487744 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 347312024 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14668562895360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137073872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14668562895360 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 138976761 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _runtime‚7772.641630716 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137073874 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137073874 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚307817942360576 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 210258216 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚310614960623616 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 210238152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14923083957760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚854 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚293149361462272 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚10399 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 210238152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160ÖÝê2†&- _wandbruntime‚7772  _runtime‚7772+HnåU‡&Ï #loss/total_loss‚46.683401299485 epoch‚8 0loss/FRt2/assignment_loss‚2.8225626988070354 2loss/SRqqt4/assignment_loss‚1.6295515197018782 0loss/FBt4/assignment_loss‚0.4379861040506512 /loss/FRt1/detection_loss‚2.7872049272060395 1loss/SRqqt2/detection_loss‚1.4026577167890288 /loss/SRqqt3/detection_loss‚1.15236978456378 /loss/FRt2/detection_loss‚2.2989233412912915 /loss/FBt1/detection_loss‚2.5824766840253557 /loss/FBt2/detection_loss‚2.9522764325141906 % _timestamp‚1.7884151622307673e+09 _runtime‚7772.642806261 1loss/SRqqt1/assignment_loss‚2.204751689732075 1loss/SRqqt4/detection_loss‚1.7798224833276537 /loss/FBt3/detection_loss‚0.8256039656594013 _step‚855 0loss/FBt1/assignment_loss‚0.4427230224220174 2loss/SRqqt2/assignment_loss‚1.7410744530959177 2loss/SRqqt3/assignment_loss‚1.2556796981165042 /loss/FRt3/detection_loss‚1.9758770472473568 /loss/FRt4/assignment_loss‚3.736619568338581 0loss/FBt2/assignment_loss‚0.5252183311062254 0loss/SRqqt1/detection_loss‚2.746312705675761 .loss/FBt4/detection_loss‚2.513804910911454 trainer/global_step‚10399 .loss/FRt3/assignment_loss‚1.75819910897149 /loss/FRt1/assignment_loss‚4.727636238932609 1loss/FBt3/assignment_loss‚0.18119395581663536 .loss/FRt4/detection_loss‚2.202874911182067Yú€ëZˆ&Ô /loss/FRt4/assignment_loss‚3.736619568338581 0loss/FBt2/assignment_loss‚0.5252183311062254 0loss/SRqqt1/detection_loss‚2.746312705675761 .loss/FBt4/detection_loss‚2.513804910911454 trainer/global_step‚10399 .loss/FRt3/assignment_loss‚1.75819910897149 /loss/FRt1/assignment_loss‚4.727636238932609 1loss/FBt3/assignment_loss‚0.18119395581663536 .loss/FRt4/detection_loss‚2.202874911182067 #loss/total_loss‚46.683401299485 epoch‚8 0loss/FRt2/assignment_loss‚2.8225626988070354 2loss/SRqqt4/assignment_loss‚1.6295515197018782 0loss/FBt4/assignment_loss‚0.4379861040506512 /loss/FRt1/detection_loss‚2.7872049272060395 1loss/SRqqt2/detection_loss‚1.4026577167890288 /loss/SRqqt3/detection_loss‚1.15236978456378 /loss/FRt2/detection_loss‚2.2989233412912915 /loss/FBt1/detection_loss‚2.5824766840253557 /loss/FBt2/detection_loss‚2.9522764325141906 % _timestamp‚1.7884151622307673e+09 _runtime‚7772.642806261 1loss/SRqqt1/assignment_loss‚2.204751689732075 1loss/SRqqt4/detection_loss‚1.7798224833276537 /loss/FBt3/detection_loss‚0.8256039656594013 _step‚855 0loss/FBt1/assignment_loss‚0.4427230224220174 2loss/SRqqt2/assignment_loss‚1.7410744530959177 2loss/SRqqt3/assignment_loss‚1.2556796981165042 /loss/FRt3/detection_loss‚1.9758770472473568×Ø+G‘‰&‚(:† Æ™äÔÏ·¢™ gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.608' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3216 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾YÎú!Š&‚(:– Æ™äÔàüªœ% disk./.usageGB‚1097.8684196472168 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.sda3.out‚ 308.16796875 proc.cpu.threads‚17) disk./.usagePercent‚73.67887081548517 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 715.546875% disk.nvme0c0n1.out‚76154.94140625 disk.sda3.in‚0 disk.dm-0.in‚ 715.546875 disk.rbd3.out‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 715.546875# memory_percent‚98.4797477722168# disk.nvme0n1.out‚76154.94140625 network.sent‚ 2.5031121e+07 disk.rbd0.out‚0! disk.rbd2.out‚178.94189453125 disk.sda.out‚ 308.16796875 disk.rbd1.in‚0& proc.memory.availableMB‚ 249.078125 network.recv‚ 2.871215e+06 disk.sda2.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 4.048828125# proc.memory.rssMB‚ 2879.48828125 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd1777.out‚0) proc.memory.percent‚17.57500171661377 cpu‚25.37454996436933 disk.dm-0.out‚76154.94140625 disk.dm-1.in‚0 disk.dm-1.out‚ 308.16796875 disk.sda2.in‚0ŒÃ²H‹&‚(:… Õ™äԨ皙 gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.237' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0474 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08{¤“%Œ&‚(:š Õ™äԆ߅ disk.sda1.out‚0# proc.memory.rssMB‚ 2879.48828125 cpu‚25.373923010910005 disk.drbd1777.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.sda.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 715.546875 disk.sda3.in‚0 disk.sda.out‚ 308.828125 disk.rbd1.out‚ 4.048828125# disk.dm-0.out‚76254.32470703125! disk.rbd2.out‚179.19189453125( proc.memory.availableMB‚ 249.08203125 network.sent‚ 2.51401e+07 disk.rbd2.in‚0 disk.nvme0c0n1.in‚ 715.546875$ memory_percent‚98.47972393035889 disk.sda1.in‚0( disk./.usagePercent‚73.6788703034701% disk./.usageGB‚1097.8684120178223 disk.dm-0.in‚ 715.546875 disk.dm-1.out‚ 308.828125 network.recv‚ 2.878336e+06& disk.nvme0n1.out‚76254.32470703125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚76254.32470703125 disk.sda3.out‚ 308.828125 disk.rbd1.in‚0 disk.sda2.in‚0 disk.drbd1777.out‚0) proc.memory.percent‚17.57500171661377Å›îI2&- _wandbruntime‚7807  _runtime‚7807T¸©^ÈŽ& _step‚856 & lr-AdamW/pg1‚5.564814473385826e-06 & lr-AdamW/pg2‚5.564814473385826e-06 trainer/global_step‚10449 % _timestamp‚1.7884151969273484e+09 _runtime‚7807.339471928ô™V¹Í&Ç _step‚856 & lr-AdamW/pg1‚5.564814473385826e-06 & lr-AdamW/pg2‚5.564814473385826e-06 trainer/global_step‚10449 % _timestamp‚1.7884151969273484e+09 _runtime‚7807.339471928Ø:³¯2&- _wandbruntime‚7807  _runtime‚7807Á»³‚&‘&èO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚294485578337280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218973184 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚294485560334336 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚309222419403776 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137694163 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14992835857408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚309222232400896 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14736841066496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚327024554884096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14694371121220 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137694163 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚309222419403776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚857 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚280138515422208 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14992616884224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 211211067 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 348925270 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137694165 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚857 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚280138515422208 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14992616884224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 211211067 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 348925270 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137694165 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚309222232400896 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14736841066496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚327024554884096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14694371121220 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137694163 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚309222419403776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 348957742 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 348937676 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137706824 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚312060608034816 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14695595164360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 348937676 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 211230854 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚309251081920000 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚294512829699072 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14738234217984 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _runtime‚7808.014696193 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14738065204736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59236450 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14738234217984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59235938 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80381231 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚294860218890516 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚312060631974912 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14994033823232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 348957742 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137706822 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _step‚858 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14738065204736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚280164441565184 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚327054880563712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14994248588800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚280164458604544 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚309250894903808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚294512847702016 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139617683 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚294512829699072 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚309251081920000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 211250918 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14695760285972 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139617169 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80381233 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚10449 % _timestamp‚1.7884151976018617e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 211250918rØbRý&•&÷M VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14738065204736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚280164441565184 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚327054880563712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14994248588800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚280164458604544 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚309250894903808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚294512847702016 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139617683 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚294512829699072 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚309251081920000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 211250918 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14695760285972 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139617169 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80381233 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚10449 % _timestamp‚1.7884151976018617e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 211250918 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 348957742 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 348937676 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137706824 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚312060608034816 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14695595164360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 348937676 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 211230854 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚309251081920000 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚294512829699072 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14738234217984 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _runtime‚7808.014696193 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14738065204736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59236450 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14738234217984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59235938 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80381231 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚294860218890516 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚312060631974912 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14994033823232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 348957742 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137706822 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _step‚858Úb0Ö2–&- _wandbruntime‚7808  _runtime‚7808ù»®[—&Õ 1loss/SRqqt2/assignment_loss‚1.342009646289829 1loss/SRqqt2/detection_loss‚1.6387348782215545 1loss/FBt4/assignment_loss‚0.33950889232358655 %loss/total_loss‚49.75140115496585 /loss/FRt4/assignment_loss‚4.164213597774506 2loss/SRqqt3/assignment_loss‚1.5280843565706164 /loss/FRt2/detection_loss‚2.3874941815932593 /loss/FRt4/detection_loss‚2.7784454694816043 trainer/global_step‚10449 0loss/FBt2/assignment_loss‚0.5489559482250895 /loss/FRt1/detection_loss‚3.3008795008063316 epoch‚8 2loss/SRqqt4/assignment_loss‚1.8635185369343128 1loss/FBt1/assignment_loss‚0.39558912230482485 /loss/FRt3/detection_loss‚1.8061751124483567 _step‚859 0loss/FRt2/assignment_loss‚3.2229927542308965 1loss/SRqqt4/detection_loss‚1.8964844177354057 -loss/FBt1/detection_loss‚2.59141338297299 /loss/FBt2/detection_loss‚3.2063645209584917 _runtime‚7808.015914108 % _timestamp‚1.7884151976038823e+09 0loss/FRt3/assignment_loss‚2.3236015792246216 1loss/FBt3/assignment_loss‚0.33212616395923306 /loss/SRqqt1/detection_loss‚3.16753913462162 1loss/SRqqt3/detection_loss‚1.3960512189078145 .loss/FBt3/detection_loss‚0.771820630809137 /loss/FRt1/assignment_loss‚3.961274037758509 2loss/SRqqt1/assignment_loss‚2.0930619023740293 /loss/FBt4/detection_loss‚2.6950621684392293Jk¬°µ˜&Ú 1loss/FBt4/assignment_loss‚0.33950889232358655 %loss/total_loss‚49.75140115496585 /loss/FRt4/assignment_loss‚4.164213597774506 2loss/SRqqt3/assignment_loss‚1.5280843565706164 /loss/FRt2/detection_loss‚2.3874941815932593 /loss/FRt4/detection_loss‚2.7784454694816043 trainer/global_step‚10449 0loss/FBt2/assignment_loss‚0.5489559482250895 /loss/FRt1/detection_loss‚3.3008795008063316 epoch‚8 2loss/SRqqt4/assignment_loss‚1.8635185369343128 1loss/FBt1/assignment_loss‚0.39558912230482485 /loss/FRt3/detection_loss‚1.8061751124483567 _step‚859 0loss/FRt2/assignment_loss‚3.2229927542308965 1loss/SRqqt4/detection_loss‚1.8968h³E«4844177354057 -loss/FBt1/detection_loss‚2.59141338297299 /loss/FBt2/detection_loss‚3.2063645209584917 _runtime‚7808.015914108 % _timestamp‚1.7884151976038823e+09 0loss/FRt3/assignment_loss‚2.3236015792246216 1loss/FBt3/assignment_loss‚0.33212616395923306 /loss/SRqqt1/detection_loss‚3.16753913462162 1loss/SRqqt3/detection_loss‚1.3960512189078145 .loss/FBt3/detection_loss‚0.771820630809137 /loss/FRt1/assignment_loss‚3.961274037758509 2loss/SRqqt1/assignment_loss‚2.0930619023740293 /loss/FBt4/detection_loss‚2.6950621684392293 1loss/SRqqt2/assignment_loss‚1.342009646289829 1loss/SRqqt2/detection_loss‚1.6387348782215545Û/ÑB‘™&‚(:† ä™äÔ™èš gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.899' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3798 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g?M,š&‚(:¡ ä™äÔ˜Ðàœ proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 715.546875 disk.sda3.in‚0 disk.nvme0n1.in‚ 715.546875 network.sent‚ 2.5145453e+07) disk./.usagePercent‚73.67887004746257$ disk./.usageGB‚1097.868408203125 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 311.36328125& disk.nvme0n1.out‚76259.03173828125 disk.rbd0.in‚0 cpu‚25.37327326993019 disk.rbd3.out‚0 disk.sda3.out‚ 311.36328125 disk.rbd1.in‚0( proc.memory.availableMB‚ 249.30078125 disk.rbd3.in‚0 disk.sda1.out‚0 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2879.48828125 disk.rbd2.in‚0 disk.sda.in‚0 disk.drbd1777.in‚0 disk.rbd1.out‚ 4.048828125# disk.dm-0.out‚76259.03173828125) proc.memory.percent‚17.57500171661377 disk.dm-1.in‚0 disk.dm-1.out‚ 311.36328125 network.recv‚ 2.882372e+06( disk.nvme0c0n1.out‚76259.03173828125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 715.546875! disk.rbd2.out‚179.26220703125$ memory_percent‚98.47838878631592¿ T‘›&‚(:† ó™äÔçî™ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.277' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4554 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾CWŒ&œ&‚(:› ó™äÔ‡Õøœ disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 313.1484375 disk.rbd1.out‚ 4.048828125% disk.nvme0n1.out‚76363.7353515625) proc.memory.percent‚17.57500171661377 disk.rbd0.in‚0' disk.nvme0c0n1.out‚76363.7353515625" disk.dm-0.out‚76363.7353515625% proc.memory.availableMB‚ 248.78125 proc.cpu.threads‚17 network.sent‚ 2.5253697e+07 network.recv‚ 2.88869e+06! disk.rbd2.out‚179.26220703125 disk.drbd1777.in‚0% disk./.usageGB‚1097.8682098388672 disk.rbd2.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 715.5546875 cpu‚25.37265468215224 disk.sda3.out‚ 313.1484375 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd1777.out‚0$ memory_percent‚98.48155975341797 disk.dm-1.out‚ 313.1484375 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 715.5546875 disk.rbd1.in‚0) disk./.usagePercent‚73.67885673507078 disk.rbd3.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 715.5546875# proc.memory.rssMB‚ 2879.48828125 disk.sda2.out‚0 disk.sda3.in‚0—¢$ü2&- _wandbruntime‚7842  _runtime‚7842,âÈž& _runtime‚7842.692296131 _step‚860 & lr-AdamW/pg1‚5.236766399614077e-06 & lr-AdamW/pg2‚5.236766399614077e-06 trainer/global_step‚10499 % _timestamp‚1.7884152322800891e+09Çß5uÍŸ&Ç & lr-AdamW/pg1‚5.236766399614077e-06 & lr-AdamW/pg2‚5.236766399614077e-06 trainer/global_step‚10499 % _timestamp‚1.7884152322800891e+09 _runtime‚7842.692296131 _step‚860ÜÓ4Ùð2 &- _wandbruntime‚7842  _runtime‚7842^6Rzî'¡&èO QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138327113 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚328540920992768 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚313477366437888 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14806343389184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14763842939628 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10499 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚296199335317872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚310655371960320 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59503015 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚310655558963200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 140245179 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14806512389120 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15063791176704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 212223807 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14806343389184 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 212203769 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14763677818224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14806512389120 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138327115 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 140245721 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚295849046574080 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚310655558963200 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚295849046574080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚328541157614592 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 212223807 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 350550922 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884152322808957e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚281435657499648 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80742706 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚296199517478636 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138327115 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚310655371960320 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚281435674539008 _runtime‚7842.693313979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 212203769 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚313477342497792 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15063578494976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138327113 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59502475 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚295849028571136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80742704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408Ðx}»ó'¢&íO _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14806512389120 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138327115 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 140245721 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚295849046574080 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚310655558963200 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚295849046574080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚328541157614592 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 212223807 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 350550922 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884152322808957e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚281435657499648 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80742706 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚296199517478636 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138327115 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚310655371960320 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚281435674539008 _runtime‚7842.693313979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 212203769 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚313477342497792 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15063578494976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138327113 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59502475 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚295849028571136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80742704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138327113 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚328540920992768 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚313477366437888 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14806343389184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14763842939628 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10499 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚296199335317872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚310655371960320 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59503015 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚310655558963200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 140245179 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14806512389120 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15063791176704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 212223807 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14806343389184 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 212203769 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14763677818224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ÝwH ¨2£&- _wandbruntime‚7843  _runtime‚7843e_`³z¤&òM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚328571490255360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚862 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚310684221479424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59507847 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15065210869248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 212243620 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80750083 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚295876315938816 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14807736527360 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138339772 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138339772 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 212243620 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚281461617721344 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 350583394 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138339774 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 212223556 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚296226849825724 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚295876297935872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 350583394 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚10499 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14807736527360 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140257928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚295876297935872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14765066982768 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚281461600681984 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 350563328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14765232104380 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚310684221479424 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚328571255744000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 Vé`qW~IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59508367 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15065000297984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 % _timestamp‚1.7884152331749103e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 140258450 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚313506279386112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚295876315938816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 350563328 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80750083 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚295876315938816 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14807736527360 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138339772 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138339772 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 212243620 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚281461617721344 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 350583394 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138339774 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 212223556 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚296226849825724 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚295876297935872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 350583394 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚10499 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14807736527360 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140257928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚295876297935872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14765066982768 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚281461600681984 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 350563328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14765232104380 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚310684221479424 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚328571255744000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59508367 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15065000297984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 % _timestamp‚1.7884152331749103e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 140258450 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚313506279386112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚295876315938816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 350563328 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064ÞëÑ‹w2¦&- _wandbruntime‚7843  _runtime‚7843º=*6Z§&Ô 0loss/FRt3/assignment_loss‚1.8184312452440676 1loss/SRqqt2/assignment_loss‚1.618101238985272 0loss/FBt1/assignment_loss‚0.4694659807708896 /loss/FRt2/detection_loss‚2.2765325709529547 /loss/FBt1/detection_loss‚2.7327247645173753 0loss/FBt4/assignment_loss‚0.4550467405053382 0loss/SRqqt1/detection_loss‚2.407351528897005 epoch‚8 _runtime‚7843.588949141 0loss/FRt4/assignment_loss‚2.8668534293466683 1loss/FBt3/assignment_loss‚0.27838474599240615 1loss/SRqqt2/detection_loss‚1.5509502252235132 .loss/FBt2/detection_loss‚2.917085886001587 /loss/FBt3/detection_loss‚0.8250864894349383 .loss/FBt4/detection_loss‚2.230071447789669 _step‚863 /loss/FRt1/assignment_loss‚5.420845937728882 2loss/SRqqt1/assignment_loss‚1.9622388035058975 .loss/FRt1/detection_loss‚2.853848985830943 /loss/FRt3/detection_loss‚1.9007814526557922 %loss/total_loss‚45.54270555778777 2loss/SRqqt3/assignment_loss‚1.3636499006428162 2loss/SRqqt4/assignment_loss‚1.4013593005947769 /loss/FRt4/detection_loss‚2.2566822882817714 1loss/SRqqt3/detection_loss‚1.2882784780002001 % _timestamp‚1.7884152331767526e+09 trainer/global_step‚10499 0loss/FBt2/assignment_loss‚0.3728106266888871 1loss/SRqqt4/detection_loss‚1.6874417729143585 /loss/FRt2/assignment_loss‚2.588681717281756‡/(Ý_¨&Ù 0loss/SRqqt1/detection_loss‚2.407351528897005 epoch‚8 _runtime‚7843.588949141 0loss/FRt4/assignment_loss‚2.8668534293466683 1loss/FBt3/assignment_loss‚0.27838474599240615 1loss/SRqqt2/detection_loss‚1.5509502252235132 .loss/FBt2/detection_loss‚2.917085886001587 /loss/FBt3/detection_loss‚0.8250864894349383 .loss/FBt4/detection_loss‚2.230071447789669 _step‚863 /loss/FRt1/assignment_loss‚5.420845937728882 2loss/SRqqt1/assignment_loss‚1.9622388035058975 .loss/FRt1/detection_loss‚2.853848985830943 /loss/FRt3/detection_loss‚1.9007814526557922 %loss/total_loss‚45.54270555778777 2loss/SRqqt3/assignment_loss‚1.3636499006428162 2loss/SRqqt4/assignment_loss‚1.4013593005947769 /loss/FRt4/detection_loss‚2.2566822882817714 1loss/SRqqt3/detection_loss‚1.2882784780002001 % _timestamp‚1.7884152331767526e+09 trainer/global_step‚10499 0loss/FBt2/assignment_loss‚0.3728106266888871 1loss/SRqqt4/detection_loss‚1.6874417729143585 /loss/FRt2/assignment_loss‚2.588681717281756 0loss/FRt3/assignment_loss‚1.8184312452440676 1loss/SRqqt2/assignment_loss‚1.618101238985272 0loss/FBt1/assignment_loss‚0.4694659807708896 /loss/FRt2/detection_loss‚2.2765325709529547 /loss/FBt1/detection_loss‚2.7327247645173753 0loss/FBt4/assignment_loss‚0.4550467405053382ß|YV‘©&‚(:† ‚šäÔÇõÉ™ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.701' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9402 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rñ*ª&‚(:Ÿ ‚šäÔåÊ㜠network.recv‚ 2.893678e+06% disk.nvme0n1.out‚76462.9111328125 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0" disk.dm-0.out‚76462.9111328125 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda.out‚ 313.6640625 network.sent‚ 2.5259942e+07% disk./.usageGB‚1097.8697052001953 disk.dm-1.out‚ 313.6640625! disk.nvme0c0n1.in‚ 715.5546875 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 4.048828125' proc.memory.availableMB‚ 249.0078125! disk.rbd2.out‚179.41552734375) proc.memory.percent‚17.57500171661377 cpu‚25.372040781857077 proc.cpu.threads‚17 disk.nvme0n1.in‚ 715.5546875 disk.sda3.in‚0$ memory_percent‚98.48017692565918# proc.memory.rssMB‚ 2879.48828125 disk.rbd0.out‚0) disk./.usagePercent‚73.67895709002421 disk.dm-0.in‚ 715.5546875 disk.rbd3.out‚0' disk.nvme0c0n1.out‚76462.9111328125 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 313.6640625K½BL«&‚(:„ ‘šäÔêÕ™ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.43' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.486 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ɾ2$¬&‚(:™ ‘šäÔ¶ù¹œ% disk./.usageGB‚1097.8698654174805 disk.drbd1777.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2879.48828125! disk.rbd2.out‚179.41552734375 disk.rbd3.in‚0' disk.nvme0c0n1.out‚76464.4443359375 network.recv‚ 2.900751e+06) disk./.usagePercent‚73.67896784234065 disk.rbd2.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 715.5546875 disk.rbd0.out‚0$ memory_percent‚98.48182201385498( proc.memory.availableMB‚ 248.73828125! disk.nvme0c0n1.in‚ 715.5546875 disk.sda2.out‚0 disk.sda.out‚ 314.40625 disk.drbd1777.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚76464.4443359375 disk.dm-1.out‚ 314.40625) proc.memory.percent‚17.57500171661377 network.sent‚ 2.536888e+07 disk.sda3.out‚ 314.40625 disk.sda.in‚0 disk.rbd1.out‚ 4.048828125 disk.nvme0n1.in‚ 715.5546875 cpu‚25.371430200315157 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda1.out‚0" disk.dm-0.out‚76464.4443359375 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda3.in‚0冘j™­&‚(:Ž  šäÔÜÓ¿™ gpu.0.gpu‚6.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.73' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.746000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›ó4D+®&‚(:   šäÔ˜Ôµ› disk.rbd0.in‚0% disk.nvme0n1.out‚76563.5263671875) proc.memory.percent‚17.57500171661377' disk.nvme0c0n1.out‚76563.5263671875 disk.sda.in‚0' proc.memory.availableMB‚ 249.2109375 disk.drbd1777.out‚0 network.recv‚ 2.903771e+06 disk.dm-1.out‚ 314.75390625 disk.rbd0.out‚0! disk.rbd2.out‚179.47802734375 disk.sda2.in‚0 disk.sda.out‚ 314.75390625# proc.memory.rssMB‚ 2879.48828125" disk.dm-0.out‚76563.5263671875 disk.rbd3.out‚0 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.134765625$ memory_percent‚98.47893714904785 cpu‚25.37082681172221% disk./.usageGB‚1096.8084716796875 disk.rbd3.in‚0( disk./.usagePercent‚73.6077368180287 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 314.75390625 proc.cpu.threads‚17 network.sent‚ 2.5373283e+07! disk.nvme0c0n1.in‚ 715.5546875 disk.dm-0.in‚ 715.5546875 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 715.55468756§¬2¯&- _wandbruntime‚7878  _runtime‚7878ŠG)€È°& & lr-AdamW/pg2‚4.918154739423255e-06 trainer/global_step‚10549 % _timestamp‚1.7884152680925725e+09 _runtime‚7878.504731875 _step‚864 & lr-AdamW/pg1‚4.918154739423255e-06¦0‚tͱ&Ç & lr-AdamW/pg1‚4.918154739423255e-06 & lr-AdamW/pg2‚4.918154739423255e-06 trainer/global_step‚10549 % _timestamp‚1.7884152680925725e+09 _runtime‚7878.504731875 _step‚864àÉ(TJ2²&- _wandbruntime‚7878  _runtime‚7878öE­í'³&çO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚297565966253080 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 352176574 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚330057547163648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138960065 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14876014711808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _runtime‚ 7878.50568456 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚312088511519744 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚314923013849088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚297212496807936 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 140884871 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚282732833655808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚312088698522624 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14833314758036 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14876183711744 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 _step‚865 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚297212514810880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138960065 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59773860 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚297566148413844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 352156534 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚312088698522624 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 140885416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 213216509 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 % _timestamp‚1.7884152680933404e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 213216509 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚297212496807936 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚10549 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138960063 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 213196471 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _runtime‚ 7878.50568456 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚312088511519744 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚314923013849088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚297212496807936 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 140884871 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚282732833655808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚312088698522624 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14833314758036 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14876183711744 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 _step‚865 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚297212514810880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138960065 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59773860 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚297566148413844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217áââ^B2µ&- _wandbruntime‚7879  _runtime‚7879òwÛ¯ø&¶&òM PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 140897943 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚314951926797312 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚282758776838144 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 213236322 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15136186533888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59779010 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140897447 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59778516 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14877576863232 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 352188980 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 352209046 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14877407849984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 352209046 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚312117361038848 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _step‚866 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚312117361038848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚312117174022656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138972722 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14877407849984 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14877576863232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138972722 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚297239766172672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81118931 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 213216258 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 _runtime‚7879.174470236 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚494 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚297239766172672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 213236322 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15135984351232 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚297593480760932 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213216258 P>Úý&·&÷M B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 213216258 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 _runtime‚7879.174470236 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚494 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚297239766172672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 213236322 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15135984351232 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚297593480760932 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213216258 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140897447 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59778516 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14877576863232 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 352188980 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 352209046 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14877407849984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 352209046 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚312117361038848 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _step‚866 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚312117361038848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚312117174022656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138972722 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14877407849984 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14877576863232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138972722 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚297239766172672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81118931 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199â`<2¸&- _wandbruntime‚7879  _runtime‚7879MWšk[¹&Õ _step‚867 2loss/SRqqt2/assignment_loss‚1.4828470690469993 2loss/SRqqt4/assignment_loss‚1.4174466922271007 0loss/SRqqt3/detection_loss‚0.938624068784217 /loss/FBt4/detection_loss‚2.5035711973905563 0loss/FRt2/assignment_loss‚2.4461408241375073 1loss/SRqqt1/assignment_loss‚2.266398845748468 /loss/FRt4/detection_loss‚1.8862854880199098 .loss/FBt1/detection_loss‚2.448680132627487 /loss/FRt1/assignment_loss‚4.841664380497402 1loss/SRqqt2/detection_loss‚1.4495483566271632 /loss/FBt2/detection_loss‚3.2074582278728485 _runtime‚7879.175912744 0loss/FBt1/assignment_loss‚0.4775959834028072 0loss/FBt4/assignment_loss‚0.3919728743082184 /loss/FRt1/detection_loss‚3.5107614199320474 &loss/total_loss‚45.127281495593415 0loss/FRt4/assignment_loss‚2.6576397471260607 0loss/FRt3/assignment_loss‚1.9714639372295804 0loss/FBt2/assignment_loss‚0.2619475705369041 .loss/FRt2/detection_loss‚2.481094539568231 1loss/SRqqt1/detection_loss‚2.7971128563989294 epoch‚8 0loss/FBt3/assignment_loss‚0.2090702311596153 /loss/FRt3/detection_loss‚1.8847747622264757 trainer/global_step‚10549 2loss/SRqqt3/assignment_loss‚1.3543068955341975 1loss/SRqqt4/detection_loss‚1.6285602882200358 /loss/FBt3/detection_loss‚0.6123151069706572 $ _timestamp‚1.788415268763869e+09¿±+0`º&Ú 0loss/FRt3/assignment_loss‚1.9714639372295804 0loss/FBt2/assignment_loss‚0.2619475705369041 .loss/FRt2/detection_loss‚2.481094539568231 1loss/SRqqt1/detection_loss‚2.7971128563989294 epoch‚8 0loss/FBt3/assignment_loss‚0.2090702311596153 /loss/FRt3/detection_loss‚1.8847747622264757 trainer/global_step‚10549 2loss/SRqqt3/assignment_loss‚1.3543068955341975 1loss/SRqqt4/detection_loss‚1.6285602882200358 /loss/FBt3/detection_loss‚0.6123151069706572 $ _timestamp‚1.788415268763869e+09 _step‚867 2loss/SRqqt2/assignment_loss‚1.4828470690469993 2loss/SRqqt4/assignment_loss‚1.4174466922271007 0loss/SRqqt3/detection_loss‚0.938624068784217 /loss/FBt4/detection_loss‚2.5035711973905563 0loss/FRt2/assignment_loss‚2.4461408241375073 1loss/SRqqt1/assignment_loss‚2.266398845748468 /loss/FRt4/detection_loss‚1.8862854880199098 .loss/FBt1/detection_loss‚2.448680132627487 /loss/FRt1/assignment_loss‚4.841664380497402 1loss/SRqqt2/detection_loss‚1.4495483566271632 /loss/FBt2/detection_loss‚3.2074582278728485 _runtime‚7879.175912744 0loss/FBt1/assignment_loss‚0.4775959834028072 0loss/FBt4/assignment_loss‚0.3919728743082184 /loss/FRt1/detection_loss‚3.5107614199320474 &loss/total_loss‚45.127281495593415 0loss/FRt4/assignment_loss‚2.6576397471260607ã€[úÊœ»&‚(:‘ ¯šäÔñœÊ™ gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.433' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.486600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Sd A ¼&‚(:• ¯šäÔÕÖ¹œ disk.dm-0.in‚ 715.5546875 disk.dm-1.in‚0 disk.dm-1.out‚ 315.29296875 disk.rbd3.in‚0 network.recv‚ 2.907816e+06) disk./.usagePercent‚73.60769534480815! disk.nvme0c0n1.in‚ 715.5546875# disk.nvme0c0n1.out‚ 76730.453125 disk.sda2.out‚0 disk.sda.out‚ 315.29296875 disk.sda1.in‚0# proc.memory.rssMB‚ 2879.48828125! disk.rbd2.out‚179.47802734375 disk.rbd2.in‚0 disk.sda3.out‚ 315.29296875) proc.memory.percent‚17.57500171661377 disk.nvme0n1.in‚ 715.5546875! disk.nvme0n1.out‚ 76730.453125 disk.sda.in‚0 disk.sda1.out‚0 network.sent‚ 2.5378812e+07% disk./.usageGB‚1096.8078536987305 disk.dm-0.out‚ 76730.453125 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0 disk.drbd1777.in‚0$ memory_percent‚98.48036766052246 cpu‚25.37021184751128 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.rbd1.out‚ 4.134765625' proc.memory.availableMB‚ 248.9765625 disk.sda2.in‚0ï)ßð½&‚(:„ ¾šäÔƦ™ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚121.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.244 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V¦m¾&‚(:œ ¾šäÔò¨Ê› disk.sda2.out‚0 disk.sda3.out‚ 315.84375# disk.dm-0.out‚76730.82177734375! disk.rbd2.out‚179.54443359375 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 715.5546875 disk.dm-1.in‚0 disk.dm-1.out‚ 315.84375 disk.rbd3.out‚0# memory_percent‚98.4790563583374( proc.memory.availableMB‚ 249.19140625 disk.sda.out‚ 315.84375 disk.drbd1777.in‚0# proc.memory.rssMB‚ 2879.48828125! disk.nvme0c0n1.in‚ 715.5546875 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0) proc.memory.percent‚17.57500171661377øÐŒmº network.sent‚ 2.5487929e+07( disk.nvme0c0n1.out‚76730.82177734375 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.197265625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 715.5546875 cpu‚25.369643463038557% disk./.usageGB‚1096.8057250976562 disk.sda3.in‚0 disk.sda1.in‚0& disk.nvme0n1.out‚76730.82177734375) disk./.usagePercent‚73.60755249260404 disk.rbd2.in‚0 network.recv‚ 2.915765e+06<X2¿&- _wandbruntime‚7914  _runtime‚7914uËN¹ÈÀ& _runtime‚ 7914.3697254 _step‚868 ' lr-AdamW/pg1‚4.6090458910057784e-06 ' lr-AdamW/pg2‚4.6090458910057784e-06 trainer/global_step‚10599 % _timestamp‚1.7884153039575667e+09ðÒ¿ªÍÁ&Ç % _timestamp‚1.7884153039575667e+09 _runtime‚ 7914.3697254 _step‚868 ' lr-AdamW/pg1‚4.6090458910057784e-06 ' lr-AdamW/pg2‚4.6090458910057784e-06 trainer/global_step‚10599äOì„[2Â&- _wandbruntime‚7914  _runtime‚7914áˆFùî'Ã&èO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚7914.370621017 _step‚869 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚313521651079168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚298932597188288 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 353782186 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚313521838082048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 trainer/global_step‚10599 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 141525081 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60044675 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚316368661260288 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331574158710272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚331574397429248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 214209211 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚298575965044736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15205521390080 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚284029992772608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14945686034432 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14945855034368 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14945855034368 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15205736168960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14902786576444 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚313521651079168 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139593013 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 % _timestamp‚1.7884153039583297e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚313521838082048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 214189173 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14902621455040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 214209211 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚298575965044736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚553 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139593015 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064“éîó'Ä&íO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331574158710272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚331574397429248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 214209211 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚298575965044736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15205521390080 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚284029992772608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14945686034432 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14945855034368 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14945855034368 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15205736168960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14902786576444 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚313521651079168 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139593013 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 % _timestamp‚1.7884153039583297e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚313521838082048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 214189173 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14902621455040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 214209211 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚298575965044736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚553 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139593015 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚7914.370621017 _step‚869 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚313521651079168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚298932597188288 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 353782186 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚313521838082048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 trainer/global_step‚10599 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 141525081 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60044675 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚316368661260288 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141537242 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 $ _timestamp‚1.788415304664188e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚284055935954944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 353814632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141537797 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81487781 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚298960111696140 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 214208960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81487783 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 353834698 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 353834698 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚331604726664192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚298959929535168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚555 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚284055918915584 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139605674 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139605674 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 214229024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚331604490055680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15207152455680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚313550500598272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 214208960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14947079172608 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14947079172608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 353814632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 _runtime‚7915.076806336 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15206939787264 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 139605672 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚298603252412416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 214229024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60049461 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14904010619584 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚316397574208512 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚298603252412416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚313550313582080 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 353834698 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚331604726664192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚298959929535168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚555 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚284055918915584 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139605674 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139605674 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 214229024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚331604490055680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15207152455680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚313550500598272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 214208960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14947079172608 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14947079172608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 353814632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 _runtime‚7915.076806336 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15206939787264 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 139605672 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚298603252412416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 214229024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60049461 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14904010619584 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚316397574208512 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚298603252412416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚313550313582080 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141537242 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 $ _timestamp‚1.788415304664188e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992ææÒÂ-2È&- _wandbruntime‚7915  _runtime‚7915 ìDW\É&Ö /loss/FRt1/assignment_loss‚4.854009012381236 0loss/FRt2/assignment_loss‚2.9278256421287856 2loss/SRqqt4/assignment_loss‚1.8582837432002028 .loss/FRt4/detection_loss‚2.086057381148924 1loss/SRqqt3/detection_loss‚1.6784602105617523 $loss/total_loss‚47.0016242951204 /loss/FRt1/detection_loss‚2.9069894194602965 1loss/SRqqt2/detection_loss‚1.6364394183906297 _step‚871 1loss/FBt1/assignment_loss‚0.22113292389379544 /loss/FRt3/detection_loss‚1.7430221255657807 .loss/FBt2/detection_loss‚3.134970853726069 % _timestamp‚1.7884153046658854e+09 /loss/FRt3/assignment_loss‚2.161372554752062 1loss/SRqqt1/assignment_loss‚2.303299827294217 trainer/global_step‚10599 /loss/FRt4/assignment_loss‚3.404044824734069 2loss/SRqqt3/assignment_loss‚1.3730620995841243 1loss/FBt3/assignment_loss‚0.17229079723915952 /loss/FBt1/detection_loss‚3.1267595489819846 /loss/FBt4/detection_loss‚2.2657653433935985 2loss/SRqqt2/assignment_loss‚1.2028601739023412 1loss/FBt4/assignment_loss‚0.43121197618334656 /loss/FBt3/detection_loss‚0.8660269918637967 epoch‚8 /loss/FRt2/detection_loss‚2.3213470820337534 0loss/FBt2/assignment_loss‚0.6414359111592299 1loss/SRqqt1/detection_loss‚2.2796571569310293 1loss/SRqqt4/detection_loss‚1.4052992766102155 _runtime‚7915.077899195õâFQaÊ&Û /loss/FRt3/detection_loss‚1.7430221255657807 .loss/FBt2/detection_loss‚3.134970853726069 % _timestamp‚1.7884153046658854e+09 /loss/FRt3/assignment_loss‚2.161372554752062 1loss/SRqqt1/assignment_loss‚2.303299827294217 trainer/global_step‚10599 /loss/FRt4/assignment_loss‚3.404044824734069 2loss/SRqqt3/assignment_loss‚1.3730620995841243 1loss/FBt3/assignment_loss‚0.17229079723915952 /loss/FBt1/detection_loss‚3.1267595489819846 /loss/FBt4/detection_loss‚2.2657653433935985 2loss/SRqqt2/assignment_loss‚1.2028601739023412 1loss/FBt4/assignment_loss‚0.43121197618334656 /loss/FBt3/detection_loss‚0.8660269918637967 epoch‚8 /loss/FRt2/detection_loss‚2.3213470820337534 0loss/FBt2/assignment_loss‚0.6414359111592299 1loss/SRqqt1/detection_loss‚2.2796571569310293 1loss/SRqqt4/detection_loss‚1.4052992766102155 _runtime‚7915.077899195 /loss/FRt1/assignment_loss‚4.854009012381236 0loss/FRt2/assignment_loss‚2.9278256421287856 2loss/SRqqt4/assignment_loss‚1.8582837432002028 .loss/FRt4/detection_loss‚2.086057381148924 1loss/SRqqt3/detection_loss‚1.6784602105617523 $loss/total_loss‚47.0016242951204 /loss/FRt1/detection_loss‚2.9069894194602965 1loss/SRqqt2/detection_loss‚1.6364394183906297 _step‚871 1loss/FBt1/assignment_loss‚0.22113292389379544ç][‡Ë&‚(:… ÍšäÔëßó™ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚95.813' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.1626 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Uâ¯j%Ì&‚(:š ÍšäÔÄÔ× disk.sda2.in‚0 disk.sda3.out‚ 316.50390625 cpu‚25.369026020732996 network.sent‚ 2.5494011e+07 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda2.out‚0) disk./.usagePercent‚73.60755249260404 disk.sda1.out‚0 disk.nvme0n1.in‚ 715.5546875 disk.sda.out‚ 316.50390625 disk.drbd1777.out‚0" disk.nvme0n1.out‚ 76731.8671875 disk.dm-0.out‚ 76731.8671875! disk.rbd2.out‚179.54443359375( proc.memory.availableMB‚ 249.12109375% disk./.usageGB‚1096.8057250976562 disk.drbd1777.in‚0 disk.rbd1.in‚0$ disk.nvme0c0n1.out‚ 76731.8671875 disk.sda3.in‚0 proc.cpu.threads‚17 disk.sda.in‚0 disk.dm-1.out‚ 316.50390625# proc.memory.rssMB‚ 2879.48828125! disk.nvme0c0n1.in‚ 715.5546875 disk.dm-0.in‚ 715.5546875 disk.dm-1.in‚0 disk.rbd3.in‚0$ memory_percent‚98.47948551177979) proc.memory.percent‚17.57500171661377 network.recv‚ 2.920244e+06 disk.sda1.in‚0 disk.rbd1.out‚ 4.197265625 disk.rbd0.in‚0 disk.rbd2.in‚0Í"}x‘Í&‚(:† ÜšäÔ–œ©™ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.598' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9196 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²=?-Î&‚(:¢ ÜšäÔ×Âßœ disk.rbd0.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 317.09765625 disk.sda1.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 715.5546875 disk.sda.out‚ 317.09765625 disk.rbd1.out‚ 4.2294921875) proc.memory.percent‚17.57500171661377 disk.drbd1777.in‚0 disk.drbd1777.out‚0$ memory_percent‚98.47946166992188# proc.memory.availableMB‚249.125 disk.rbd0.out‚0! disk.rbd2.out‚179.60693359375 disk.sda3.in‚0 disk.dm-0.in‚ 715.5546875 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 715.5546875 disk.sda2.out‚0& disk.nvme0n1.out‚76733.03857421875# proc.memory.rssMB‚ 2879.48828125 cpu‚25.368399651909932( disk.nvme0c0n1.out‚76733.03857421875% disk./.usageGB‚1096.8064575195312) disk./.usagePercent‚73.60760164605063 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0# disk.dm-0.out‚76733.03857421875 disk.rbd3.in‚0 network.sent‚ 2.5602502e+07 network.recv‚ 2.927002e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 317.09765625½|2Ï&- _wandbruntime‚7949  _runtime‚7949b=ÈÐ& ' lr-AdamW/pg2‚4.3095042721823605e-06 trainer/global_step‚10649 $ _timestamp‚1.788415339347606e+09 _runtime‚ 7949.75976431 _step‚872 ' lr-AdamW/pg1‚4.3095042721823605e-06h&±èÍÑ&Ç ' lr-AdamW/pg2‚4.3095042721823605e-06 trainer/global_step‚10649 $ _timestamp‚1.788415339347606e+09 _runtime‚ 7949.75976431 _step‚872 ' lr-AdamW/pg1‚4.3095042721823605e-06èB9û2Ò&- _wandbruntime‚7949  _runtime‚7949Þøš2î'Ó&èO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 215181875 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚299939451284480 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚285327134850048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60317227 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _step‚873 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚333090683960320 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15015526356992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60317767 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚285327151889408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140225963 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚317814308671488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14972258394852 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚314954790638592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚314954790638592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚299939433281536 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 142166481 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚317814284731392 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14972093273448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚300299410284260 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15015357357056 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 355427878 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 142167023 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚299939451284480 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 215181875 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81849254 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15015526356992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚333090920582144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚300299228123496 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚299939451284480 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 215181875 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81849254 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15015526356992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚333090920582144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚300299228123496 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚314954790638592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚299939433281536 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 142166481 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚317814284731392 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14972093273448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚300299410284260 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728éõ=w‘Õ&‚(:† ëšäÔì±™ gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.872' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0  †1Ö&‚(:¦ ëšäÔ᧠œ! disk.nvme0c0n1.in‚ 715.5546875( disk.nvme0c0n1.out‚76733.75341796875 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda.in‚0& disk.nvme0n1.out‚76733.75341796875$ memory_percent‚98.47943782806396 disk.sda3.out‚ 317.50390625 disk.dm-1.in‚0( proc.memory.availableMB‚ 249.12890625) proc.memory.percent‚17.57500171661377 network.recv‚ 2.930596e+06 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 4.2294921875 disk.rbd0.out‚0! disk.rbd2.out‚179.60693359375 disk.drbd1777.in‚0# disk.dm-0.out‚76733.75341796875 disk.rbd2.in‚0 disk.sda.out‚ 317.50390625 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 317.50390625) disk./.usagePercent‚73.60760497414857 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 715.5546875 disk.dm-0.in‚ 715.5546875# proc.memory.rssMB‚ 2879.48828125 cpu‚25.367803941289715 network.sent‚ 2.560799e+07% disk./.usageGB‚1096.8065071105957QG…#2×&- _wandbruntime‚7950  _runtime‚7950JÏC%÷&Ø&ñM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15016750495232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚ 7950.43728735 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 355460350 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 215201662 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15278029611520 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15277819040256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚300326742631348 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 215221726 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚285353078032384 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140238622 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚299966702646272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚299966720649216 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14973647559604 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚333121251231232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚333121016719872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140238624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60323021 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 142179654 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142179129 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985NÖVü&Ù&öM H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚299966720649216 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15016919508480 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60322498 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15016919508480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140238624 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81856631 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142179129 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15016750495232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚ 7950.43728735 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 355460350 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 215201662 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15278029611520 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15277819040256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚300326742631348 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 215221726 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚285353078032384 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140238622 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚299966702646272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚299966720649216 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14973647559604 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚333121251231232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚333121016719872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140238624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60323021 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 142179654 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 EDeviceStatsMont^;_itor.on_train_batch_start/active.small_pool.peak‚24128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15085197679616 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚316387930198016 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15347544456704 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15041565091856 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚301302919521280 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15085028679680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚334607500539392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15085197679616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚286624293966848 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 357033490 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚301302919521280 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚566 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚316388117200896 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚301302901518336 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140858913 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60590057 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316387930198016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 216174577 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚301665859058704 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316387930198016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 216174577 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚301665859058704 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15085197679616 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚316387930198016 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15347544456704 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15041565091856 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199íñ«2ç&- _wandbruntime‚7985  _runtime‚7985’ÍÌLø&è&òM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15042954256400 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140871574 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60596310 _runtime‚7985.853203168 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚316416592700928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 216214428 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15086421817856 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚301330188886016 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15086421817856 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 142821793 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚319288869030912 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 % _timestamp‚1.7884153754405212e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140871572 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚319288845090816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15043119378012 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 357086002 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15348959568896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82225483 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚301330188886016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚316416779717120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚301693373566556 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚286650254188544 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15086590831104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 216214428 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚334637828599808 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚301330170883072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60595767 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142821248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚301693191405584 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚316416779717120 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 216194364 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140871574 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 216194364 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚301330170883072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15086590831104 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 trainer/global_step‚10699 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 357086002 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142821248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚301693191405584 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚316416779717120 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 216194364 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140871574 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 216194364 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚301330170883072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ]JDeviceStatsMonitor.on_train_batcÿT1œh_end/allocated_bytes.small_pool.allocated‚15086590831104 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 trainer/global_step‚10699 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 357086002 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140871574 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60596310 _runtime‚7985.853203168 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚316416592700928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 216214428 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15086421817856 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚301330188886016 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15086421817856 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 142821793 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚319288869030912 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 % _timestamp‚1.7884153754405212e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140871572 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚319288845090816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15043119378012 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 357086002 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15348959568896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0î›çi2ê&- _wandbruntime‚7985  _runtime‚7985ý}¡@Vë&Ð /loss/FBt1/detection_loss‚2.7444779872894287 .loss/FBt4/detection_loss‚2.483582124761913 epoch‚8 % _timestamp‚1.7884153754421806e+09 _step‚879 0loss/FRt2/assignment_loss‚2.7348315703625583 2loss/SRqqt2/assignment_loss‚1.2003143566248358 0loss/FBt4/assignment_loss‚0.5323499454523442 1loss/SRqqt2/detection_loss‚1.5441678531590055 /loss/FBt2/detection_loss‚3.2904472265924727 2loss/SRqqt3/assignment_loss‚1.5808802435640246 /loss/FRt4/detection_loss‚2.0206773739170143 0loss/SRqqt1/detection_loss‚2.654514934007938 _runtime‚7985.854241785 0loss/FRt3/assignment_loss‚2.2933362871408463 0loss/FRt4/assignment_loss‚3.4967168348806874 2loss/SRqqt1/assignment_loss‚2.1749194172712474 0loss/FBt3/assignment_loss‚0.3042705128981411 1loss/SRqqt3/detection_loss‚1.9344173077493907 /loss/FBt3/detection_loss‚0.7018785302983209 0loss/FRt1/assignment_loss‚4.6546946413376755 1loss/FBt1/assignment_loss‚0.38361180906014686 %loss/total_loss‚47.39026179349659 2loss/SRqqt4/assignment_loss‚1.5489818707254588 .loss/FBt2/assignment_loss‚0.56842042000166 ,loss/FRt1/detection_loss‚2.7730052541284 .loss/FRt3/detection_loss‚1.771942662075162 0loss/SRqqt4/detection_loss‚1.736387386041529 trainer/global_step‚10699 -loss/FRt2/detection_loss‚2.26143524415639!¡T[ì&Õ /loss/FBt1/detection_loss‚2.7444779872894287 .loss/FBt4/detection_loss‚2.483582124761913 epoch‚8 % _timestamp‚1.7884153754421806e+09 _step‚879 0loss/FRt2/assignment_loss‚2.7348315703625583 2loss/SRqqt2/assignment_loss‚1.2003143566248358 0loss/FBt4/assignment_loss‚0.5323499454523442 1loss/SRqqt2/detection_loss‚1.5441678531590055 /loss/FBt2/detection_loss‚3.2904472265924727 2loss/SRqqt3/assignment_loss‚1.5808802435640246 /loss/FRt4/detection_loss‚2.0206773739170143 0loss/SRqqt1/detection_loss‚2.654514934007938 _runtime‚7985.854241785 0loss/FRt3/assignment_loss‚2.2933362871408463 0loss/FRt4/assignment_loss‚3.4967168348806874 2loss/SRqqt1/assignment_loss‚2.1749194172712474 0loss/FBt3/assignment_loss‚0.3042705128981411 1loss/SRqqt3/detection_loss‚1.9344173077493907 /loss/FBt3/detection_loss‚0.7018785302983209 0loss/FRt1/assignment_loss‚4.6546946413376755 1loss/FBt1/assignment_loss‚0.38361180906014686 %loss/total_loss‚47.39026179349659 2loss/SRqqt4/assignment_loss‚1.5489818707254588 .loss/FBt2/assignment_loss‚0.56842042000166 ,loss/FRt1/detection_loss‚2.7730052541284 .loss/FRt3/detection_loss‚1.771942662075162 0loss/SRqqt4/detection_loss‚1.736387386041529 trainer/global_step‚10699 -loss/FRt2/detection_loss‚2.26143524415639ï(É]±í&‚(:… ˜›äԟ⢙ gpu.0.gpu‚25.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.353' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0706 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X F&0î&‚(:¥ ˜›äÔÉÏÚœ network.recv‚ 2.948771e+06 disk.nvme0n1.in‚ 715.55859375 disk.rbd3.out‚0) proc.memory.percent‚17.57500171661377 network.sent‚ 2.5809361e+07 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 715.55859375 disk.dm-1.in‚0' proc.memory.availableMB‚ 248.6640625" disk.nvme0c0n1.in‚ 715.55859375 disk.sda3.out‚ 319.24609375& disk.nvme0n1.out‚76736.64306640625 cpu‚25.366017756104913 proc.cpu.threads‚17 disk.sda.out‚ 319.24609375 disk.rbd1.in‚0$ memory_percent‚98.48227500915527 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0) disk./.usagePercent‚73.60773220989309( disk.nvme0c0n1.out‚76736.64306640625 disk.sda2.out‚0 disk.dm-1.out‚ 319.24609375 disk.rbd0.out‚0 disk.rbd2.out‚ 182.955078125 disk.rbd3.in‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.rbd1.out‚ 4.35546875 disk.rbd0.in‚0% disk./.usageGB‚1096.8084030151367# disk.dm-0.out‚76736.64306640625# proc.memory.rssMB‚ 2879.48828125vv¶Æšï&‚(: §›äÔ¸‚¯™ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚91.72' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.343999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TïßX+ð&‚(:  §›äÔÇç¼›$ disk.nvme0n1.out‚76739.455078125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda.in‚0$ memory_percent‚98.48225116729736! disk.dm-0.out‚76739.455078125) disk./.usagePercent‚73.59627177661528 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.35546875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 network.recv‚ 2.955732e+06 disk.drbd1777.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.5917678e+07" disk.nvme0c0n1.in‚ 715.55859375 disk.sda3.in‚0 disk.sda.out‚ 319.66015625( proc.memory.availableMB‚ 248.66796875 disk.dm-1.in‚0% disk./.usageGB‚1096.6376342773438 disk.dm-0.in‚ 715.55859375) proc.memory.percent‚17.57500171661377 disk.nvme0n1.in‚ 715.55859375 disk.dm-1.out‚ 319.66015625 disk.rbd2.in‚0 disk.rbd2.out‚ 184.466796875& disk.nvme0c0n1.out‚76739.455078125 disk.sda2.out‚0 disk.sda3.out‚ 319.66015625# proc.memory.rssMB‚ 2879.48828125 cpu‚25.365435815665094 µÆõ2ñ&- _wandbruntime‚8020  _runtime‚8020Á¢I§Èò& & lr-AdamW/pg2‚3.739370412609937e-06 trainer/global_step‚10749 % _timestamp‚1.7884154097218645e+09 _runtime‚8020.134008572 _step‚880 & lr-AdamW/pg1‚3.739370412609937e-06‘¢Õ®Íó&Ç & lr-AdamW/pg2‚3.739370412609937e-06 trainer/global_step‚10749 % _timestamp‚1.7884154097218645e+09 _runtime‚8020.134008572 _step‚880 & lr-AdamW/pg1‚3.739370412609937e-06ð,m*2ô&- _wandbruntime‚8020  _runtime‚8020×î'õ&èO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82586956 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60863702 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚320705603493888 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15418292222976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚287921453083648 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15154869002240 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚529 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200098816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚302666369755136 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _runtime‚8020.134989861 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 358679182 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚317821256760320 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15111036910264 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15154700002304 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143450656 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚302666369755136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 _step‚881 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15418492321792 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚317821069757440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82586954 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 % _timestamp‚1.7884154097226386e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚336123871776768 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚317821069757440 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 358659142 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224038912 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚320705579553792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚303032672154676 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141491863 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 217167279 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚303032489993912 trainer/global_step‚10749 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚527 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 217187317 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚287921470123008 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 217167279 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143451185 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60864229 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217‡(Zºó'ö&íO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143451185 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60864229 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82586956 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60863702 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚320705603493888 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15418292222976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚287921453083648 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15154869002240 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚529 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200098816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚302666369755136 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _runtime‚8020.134989861 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 358679182 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚317821256760320 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15111036910264 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15154700002304 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143450656 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚302666369755136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 _step‚881 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15418492321792 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚317821069757440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82586954 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 % _timestamp‚1.7884154097226386e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚336123871776768 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚317821069757440 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 358659142 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224038912 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚320705579553792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚303032672154676 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141491863 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 217167279 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚303032489993912 trainer/global_step‚10749 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚527 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 217187317 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚287921470123008 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 217167279ñä[ô`2÷&- _wandbruntime‚8020  _runtime‚8020¥2œÕ€ø&ñM ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15156262153728 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚287947396265984 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 YFDeviceStatsMonitor.on_train_batc=ýXLw%h_end/allocated_bytes.small_pool.freed‚15156093140480 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226122752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚336154208428544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 358691588 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82594333 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15419918109184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚317849732260352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚320734492502016 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.788415410390948e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚303059822340792 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 trainer/global_step‚10749 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82594331 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚317849919276544 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358711654 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143463409 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 217207130 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15112426074808 _runtime‚8020.803681622 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15156093140480 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚317849919276544 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 217187066 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚320734516442112 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚303060004501764 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚302693657122816 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 358711654 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _step‚882 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141504524 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚302693657122816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202182656 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60869600 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 217187066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚302693639119872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 217207130 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚287947413305344 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 50415534089¯©ü&ù&öM A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚317849919276544 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358711654 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143463409 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 217207130 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15112426074808 _runtime‚8020.803681622 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15156093140480 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚317849919276544 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 217187066 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚320734516442112 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚303060004501764 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚302693657122816 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 358711654 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _step‚882 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141504524 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚302693657122816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202182656 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60869600 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 217187066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚302693639119872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 217207130 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚287947413305344 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15156262153728 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚287947396265984 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15156093140480 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226122752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚336154208428544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 358691588 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82594333 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15419918109184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚317849732260352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚320734492502016 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.788415410390948e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚303059822340792 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 trainer/global_step‚10749 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82594331òf¯¸2ú&- _wandbruntime‚8020  _runtime‚8020 ¶<òZû&Ô 0loss/FBt2/assignment_loss‚1.0334246748825535 1loss/SRqqt2/detection_loss‚1.4973288811170138 _runtime‚8020.805090667 _step‚883 .loss/FBt1/detection_loss‚3.054531474908193 %loss/total_loss‚45.96002125346596 trainer/global_step‚10749 1loss/FBt3/assignment_loss‚0.20793904719028986 1loss/FBt4/assignment_loss‚0.38446798700843987 .loss/FRt1/detection_loss‚3.264833187063535 1loss/SRqqt4/detection_loss‚1.6158791377879025 epoch‚8 /loss/FRt1/assignment_loss‚4.250978479782741 0loss/FRt4/assignment_loss‚3.8996217449506125 /loss/FRt4/detection_loss‚2.1041263043880516 1loss/SRqqt1/detection_loss‚2.6787648797035217 1loss/SRqqt3/detection_loss‚1.1812801326022428 /loss/FBt3/detection_loss‚0.6432684178020558 1loss/SRqqt4/assignment_loss‚1.748770765175945 .loss/FRt2/detection_loss‚2.814994556062362 % _timestamp‚1.7884154103930123e+09 0loss/FRt3/assignment_loss‚2.1528271056236106 2loss/SRqqt1/assignment_loss‚2.3308877761547384 1loss/SRqqt2/assignment_loss‚1.276737234247132 -loss/FBt2/detection_loss‚2.02935016900301 .loss/FBt4/detection_loss‚1.775276746820001 2loss/SRqqt3/assignment_loss‚1.2735349556394653 1loss/FBt1/assignment_loss‚0.33120738083663226 /loss/FRt3/detection_loss‚1.7707101128836895 0loss/FRt2/assignment_loss‚2.6392801018322216ÁíP$_ü&Ù 2loss/SRqqt3/assignment_loss‚1.2735349556394653 1loss/FBt1/assignment_loss‚0.33120738083663226 /loss/FRt3/detection_loss‚1.7707101128836895 0loss/FRt2/assignment_loss‚2.6392801018322216 0loss/FBt2/assignment_loss‚1.0334246748825535 1loss/SRqqt2/detection_loss‚1.4973288811170138 _runtime‚8020.805090667 _step‚883 .loss/FBt1/detection_loss‚3.054531474908193 %loss/total_loss‚45.96002125346596 trainer/global_step‚10749 1loss/FBt3/assignment_loss‚0.20793904719028986 1loss/FBt4/assignment_loss‚0.38446798700843987 .loss/FRt1/detection_loss‚3.264833187063535 1loss/SRqqt4/detection_loss‚1.6158791377879025 epoch‚8 /loss/FRt1/assignment_loss‚4.250978479782741 0loss/FRt4/assignment_loss‚3.8996217449506125 /loss/FRt4/detection_loss‚2.1041263043880516 1loss/SRqqt1/detection_loss‚2.6787648797035217 1loss/SRqqt3/detection_loss‚1.1812801326022428 /loss/FBt3/detection_loss‚0.6432684178020558 1loss/SRqqt4/assignment_loss‚1.748770765175945 .loss/FRt2/detection_loss‚2.814994556062362 % _timestamp‚1.7884154103930123e+09 0loss/FRt3/assignment_loss‚2.1528271056236106 2loss/SRqqt1/assignment_loss‚2.3308877761547384 1loss/SRqqt2/assignment_loss‚1.276737234247132 -loss/FBt2/detection_loss‚2.02935016900301 .loss/FBt4/detection_loss‚1.775276746820001óˆR‹«Œý&‚(: ¶›äÔõ¬¹™ gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.5' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£VÎ+þ&‚(:  ¶›äÔåצœ network.sent‚ 2.5923901e+07 disk.sda.in‚0 disk.drbd1777.in‚0! disk.dm-0.out‚76739.728515625 disk.sda2.in‚0) disk./.usagePercent‚73.59627152060774$ disk.nvme0n1.out‚76739.728515625 disk.dm-0.in‚ 715.55859375 disk.dm-1.out‚ 320.05078125 disk.sda2.out‚0% disk./.usageGB‚1096.6376304626465 disk.rbd3.out‚0 disk.sda3.out‚ 320.05078125# proc.memory.rssMB‚ 2879.48828125 network.recv‚ 2.960608e+06 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.35546875 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 715.55859375& disk.nvme0c0n1.out‚76739.728515625 disk.sda3.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd3.in‚0$ memory_percent‚98.48220348358154) proc.memory.percent‚17.57500171661377 cpu‚25.364873620735235 proc.cpu.threads‚17 disk.sda1.in‚0 disk.nvme0n1.in‚ 715.55859375 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 184.630859375 disk.sda.out‚ 320.05078125( proc.memory.availableMB‚ 248.67578125çIàõ‘ÿ&‚(:† Å›äÔ‚¶Á™ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.901' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_gt€'‚(:” Å›äÔ ³å› disk.sda3.out‚ 320.671875 disk.rbd1.out‚ 4.35546875 network.sent‚ 2.6033148e+07 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 320.671875 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 715.55859375 disk.sda3.in‚0 disk.nvme0n1.in‚ 715.55859375# proc.memory.rssMB‚ 2879.48828125) proc.memory.percent‚17.57500171661377 disk.sda1.in‚0 disk.dm-0.out‚76873.69921875 disk.sda2.in‚0 cpu‚25.364291316973244$ disk./.usageGB‚1096.767333984375 disk.rbd0.in‚0 disk.drbd1777.in‚0$ memory_percent‚98.48380088806152 proc.cpu.threads‚17) disk./.usagePercent‚73.60497603277945# disk.nvme0n1.out‚76873.69921875 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.rbd2.out‚ 186.142578125 disk.rbd3.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 248.4140625 disk.sda.out‚ 320.671875 network.recv‚ 2.96827e+06 disk.dm-0.in‚ 715.55859375 disk.rbd2.in‚0 disk.rbd3.out‚0% disk.nvme0c0n1.out‚76873.699218759Å%¨›'‚(: Ô›äÔ©Þ™™ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚89.821' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.964199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹Ä-Õ&‚'‚(:› Ô›äÔ¶­šœ disk.sda3.in‚0 disk.dm-0.in‚ 715.55859375 disk.dm-0.out‚77007.12890625 cpu‚25.36370515406813 disk.rbd0.in‚0 network.sent‚ 2.6038805e+07$ disk./.usageGB‚1096.897373199463 disk.drbd1777.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 321.19140625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 321.19140625 disk.rbd1.in‚0 disk.nvme0n1.in‚ 715.55859375 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 715.55859375 disk.sda2.in‚0 disk.sda3.out‚ 321.19140625 disk.drbd1777.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚73.61370307361416 disk.sda1.in‚0# disk.nvme0n1.out‚77007.12890625 disk.rbd2.in‚0! disk.rbd2.out‚186.31494140625% disk.nvme0c0n1.out‚77007.12890625 disk.sda2.out‚0 disk.rbd1.out‚ 4.35546875) proc.memory.percent‚17.57500171661377 disk.sda.in‚0 disk.rbd0.out‚0# memory_percent‚98.4809398651123' proc.memory.availableMB‚ 248.8828125# proc.memory.rssMB‚ 2879.48828125 network.recv‚ 2.972487e+0690;2ƒ'- _wandbruntime‚8056  _runtime‚8056ð›,XÈ„' & lr-AdamW/pg2‚3.468896986902724e-06 trainer/global_step‚10799 % _timestamp‚1.7884154457038498e+09 _runtime‚8056.116204134 _step‚884 & lr-AdamW/pg1‚3.468896986902724e-06øPÍ…'Ç & lr-AdamW/pg1‚3.468896986902724e-06 & lr-AdamW/pg2‚3.468896986902724e-06 trainer/global_step‚10799 % _timestamp‚1.7884154457038498e+09 _runtime‚8056.116204134 _step‚884ô¥©&ä2†'- _wandbruntime‚8056  _runtime‚80564Òd‡'èO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚322151226964992 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15489266984960 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142124815 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15180673850076 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.curren½(¬~Ýt‚ 187002880 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15224371324928 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 144090850 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202195968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚322151250905088 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚319254209316864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61135046 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15180508728672 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218159981 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚304399120929120 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15224540324864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚319254209316864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 trainer/global_step‚10799 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚304029837991936 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚304029855994880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82955804 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 % _timestamp‚1.7884154457047644e+09 _step‚885 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142124813 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚304399303089884 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 360304834 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0&l¬Õó'ˆ'íO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _runtime‚8056.117266255 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 360284794 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15224540324864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 218180019 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚319254396319744 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144091383 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚289218612200448 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚337640720086016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚337640493949952 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142124815 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15180673850076 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15224371324928 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 144090850 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202195968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚322151250905088 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚319254209316864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61135046 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15180508728672 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218159981 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚304399120929120 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15224540324864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚319254209316864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 trainer/global_step‚10799 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚304029837991936 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚304029855994880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82955804 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 % _timestamp‚1.7884154457047644e+09 _step‚885 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142124813 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚304399303089884 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 360304834 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚322151226964992 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15489266984960 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚289244572422144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚ 8056.82359054 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚337671057622016 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144103470 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚319282871819776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15490893768704 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142137472 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚304057107356672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 360317240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82963183 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 360337306 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚337670827304960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15225933476352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚322180139913216 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚304057125359616 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61140784 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15182063014828 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15225933476352 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15181897893216 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚319283058835968 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 360337306 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884154464107208e+09 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 144103967 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82963181 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚304057107356672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚886 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚319283058835968 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚304426453276000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 trainer/global_step‚10799 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 360317240 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚304057125359616 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 218199832 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15225764463104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142137474 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142137474! ÊÔä‹'öM FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884154464107208e+09 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 144103967 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82963181 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚304057107356672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚886 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚319283058835968 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚304426453276000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 trainer/global_step‚10799 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 360317240 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚304057125359616 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 218199832 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15225764463104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142137474 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142137474 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15490687391744 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚495 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚322180163853312 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61140289 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚289244572422144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚ 8056.82359054 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚337671057622016 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144103470 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚319282871819776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15490893768704 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142137472 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚304057107356672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.o}@øK n_train_batch_end/active.all.freed‚ 360317240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82963183 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 360337306 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚337670827304960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15225933476352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚322180139913216 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚304057125359616 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61140784 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15182063014828 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15225933476352 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15181897893216 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚319283058835968 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 360337306öÜœ€2Œ'- _wandbruntime‚8056  _runtime‚8056ûK“’X'Ò .loss/FBt1/detection_loss‚2.979070223429624 .loss/FBt2/detection_loss‚3.258009148495538 /loss/FRt2/detection_loss‚2.1049018605631225 trainer/global_step‚10799 0loss/FBt4/assignment_loss‚0.5861754333184633 /loss/FRt1/assignment_loss‚4.184192834361907 0loss/FRt3/assignment_loss‚2.5282235923020737 /loss/FRt4/assignment_loss‚3.521347079012129 1loss/SRqqt4/assignment_loss‚1.469302101060748 0loss/FBt2/assignment_loss‚0.5339749067264036 /loss/FRt3/detection_loss‚2.0256411875395672 2loss/SRqqt1/assignment_loss‚2.1440274062610807 .loss/FRt1/detection_loss‚2.977856820629489 1loss/SRqqt4/detection_loss‚1.8775923829129402 &loss/total_loss‚47.466105715072686 _runtime‚8056.824942948 _step‚887 1loss/SRqqt1/detection_loss‚3.2017459784235274 1loss/SRqqt2/detection_loss‚1.4590877333367374 0loss/FRt2/assignment_loss‚2.7602704799905116 0loss/FBt3/assignment_loss‚0.2396679875341913 /loss/FBt3/detection_loss‚0.7489041312377284 .loss/FBt4/detection_loss‚2.265043560947691 2loss/SRqqt2/assignment_loss‚1.2722141836887935 0loss/FBt1/assignment_loss‚0.2572481071097975 0loss/SRqqt3/detection_loss‚1.551372070486347 epoch‚8 /loss/FRt4/detection_loss‚1.8985435294884223 $ _timestamp‚1.788415446412861e+09 2loss/SRqqt3/assignment_loss‚1.6216929762158543ÉÜó ]Ž'× /loss/FRt3/detection_loss‚2.0256411875395672 2loss/SRqqt1/assignment_loss‚2.1440274062610807 .loss/FRt1/detection_loss‚2.977856820629489 1loss/SRqqt4/detection_loss‚1.8775923829129402 &loss/total_loss‚47.466105715072686 _runtime‚8056.824942948 _step‚887 1loss/SRqqt1/detection_loss‚3.2017459784235274 1loss/SRqqt2/detection_loss‚1.4590877333367374 0loss/FRt2/assignment_loss‚2.7602704799905116 0loss/FBt3/assignment_loss‚0.2396679875341913 /loss/FBt3/detection_loss‚0.7489041312377284 .loss/FBt4/detection_loss‚2.265043560947691 2loss/SRqqt2/assignment_loss‚1.2722141836887935 0loss/FBt1/assignment_loss‚0.2572481071097975 0loss/SRqqt3/detection_loss‚1.551372070486347 epoch‚8 /loss/FRt4/detection_loss‚1.8985435294884223 $ _timestamp‚1.788415446412861e+09 2loss/SRqqt3/assignment_loss‚1.6216929762158543 .loss/FBt1/detection_loss‚2.979070223429624 .loss/FBt2/detection_loss‚3.258009148495538 /loss/FRt2/detection_loss‚2.1049018605631225 trainer/global_step‚10799 0loss/FBt4/assignment_loss‚0.5861754333184633 /loss/FRt1/assignment_loss‚4.184192834361907 0loss/FRt3/assignment_loss‚2.5282235923020737 /loss/FRt4/assignment_loss‚3.521347079012129 1loss/SRqqt4/assignment_loss‚1.469302101060748 0loss/FBt2/assignment_loss‚0.5339749067264036÷´{'‚(:… ã›äÔÑêÁ™ gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.889' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´ã'‚(:“ ã›äÔ¨èõœ disk.sda1.in‚0 disk.drbd1777.in‚0# proc.memory.rssMB‚ 2879.48828125 network.sent‚ 2.6044264e+07 disk.sda2.in‚0 disk.drbd1777.out‚0 disk.nvme0n1.out‚ 77007.78125 disk.dm-0.out‚ 77007.78125 cpu‚25.363119220336095 network.recv‚ 2.975989e+06$ memory_percent‚98.48134517669678( proc.memory.availableMB‚ 248.81640625 disk.dm-1.out‚ 321.7890625 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 715.55859375" disk.nvme0c0n1.out‚ 77007.78125 disk.sda.out‚ 321.7890625 disk.rbd1.in‚0) disk./.usagePercent‚73.59887153312549 disk.dm-1.in‚0 disk.sda3.in‚0 disk.rbd1.out‚ 4.35546875 disk.nvme0n1.in‚ 715.55859375) proc.memory.percent‚17.57500171661377 disk.rbd0.in‚0! disk.rbd2.out‚187.82666015625 disk.dm-0.in‚ 715.55859375 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1096.6763725280762 disk.rbd3.out‚0 disk.sda3.out‚ 321.7890625eêÊÄ‘'‚(:… ò›äÔØÒ™ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.0014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HìâÖ(’'‚(: ò›äÔ·•–œ disk.sda.in‚0# disk.nvme0n1.out‚77008.01953125 disk.dm-0.out‚77008.01953125 disk.dm-1.in‚0 disk.dm-1.out‚ 322.21484375 disk.sda3.in‚0# proc.memory.rssMB‚ 2879.48828125 cpu‚25.362544841025223 disk.drbd1777.in‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 715.55859375 disk.sda1.out‚0$ memory_percent‚98.48432540893555& proc.memory.availableMB‚ 248.328125) proc.memory.percent‚17.57500171661377% disk./.usageGB‚1096.6763725280762! disk.rbd2.out‚187.99462890625 disk.rbd3.in‚0 disk.sda.out‚ 322.21484375 network.recv‚ 2.983853e+06 disk.nvme0n1.in‚ 715.55859375 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.sda3.out‚ 322.21484375) disk./.usagePercent‚73.59887153312549 disk.rbd1.out‚ 4.35546875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 715.55859375 disk.sda2.in‚0 network.sent‚ 2.6153432e+07 disk.rbd2.in‚0% disk.nvme0c0n1.out‚77008.01953125µ¬G2“'- _wandbruntime‚8091  _runtime‚8091‹3KÊ”'Ä ' lr-AdamW/pg1‚3.2082283961122905e-06 ' lr-AdamW/pg2‚3.2082283961122905e-06 trainer/global_step‚10849 % _timestamp‚1.7884154815828326e+09 _runtime‚8091.994977886 _step‚888?QtÏ•'É trainer/global_step‚10849 % _timestamp‚1.7884154815828326e+09 _runtime‚8091.994977886 _step‚888 ' lr-AdamW/pg1‚3.2082283961122905e-06 ' lr-AdamW/pg2‚3.2082283961122905e-06ø)˜2–'- _wandbruntime‚8091  _runtime‚8091Ÿ»gzî'—'èO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚305393306228736 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142757765 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61407845 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚562 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230330368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚305765934025092 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15294211647488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142757763 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61408405 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142757765 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 219172721 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚305765751864328 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15294042647552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 219152683 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320687348876288 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 361930486 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15250145668484 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 361910446 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15560445211136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚323596874376192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚305393324231680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 _step‚889 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚320687348876288 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚305393324231680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚320687535879168 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144733061 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚290515788356608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15294211647488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚560 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚8091.995902148 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 45415680005ZvÒó'˜'íO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83324656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206390272 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚305393306228736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15249980547080 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚320687535879168 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144733061 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚290515788356608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15294211647488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚560 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚8091.995902148 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚305393306228736 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142757765 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61407845 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚562 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230330368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚305765934025092 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15294211647488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142757763 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61408405 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142757765 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 219172721 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚305765751864328 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15294042647552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 219152683 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320687348876288 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 361930486 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15250145668484 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 361910446 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15560445211136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚323596874376192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚305393324231680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 _step‚889 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚320687348876288 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚305393324231680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 trainer/global_step‚10849 % _timestamp‚1.7884154822368798e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83332031 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚305793266372180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 361942892 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚320716198395392 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144745397 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytYÄîrÝes.all.freed‚320716011379200 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15295604798976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 361942892 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚305420575593472 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 144745944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142770422 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142770422 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚890 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219192534 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 361962958 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚545 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142770424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15295435785728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83332033 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚305793084211208 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15295435785728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚339187680315392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚305420593596416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚323625811264512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15561869050880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 219172470 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142770424 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15251534833236 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61413911 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚290541714499584 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160WdÅý&›'÷M EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83332031 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚305793266372180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 361942892 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚320716198395392 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144745397 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚320716011379200 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15295604798976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 361942892 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚305420575593472 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 144745944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142770422 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142770422 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚890 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219192534 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 361962958 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚545 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142770424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15295435785728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83332033 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚305793084211208 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15295435785728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚339187680315392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚305420593596416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚323625811264512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15561869050880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 219172470 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142770424 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15251534833236 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61413911 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚290541714499584 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚547 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚305420593596416 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 trainer/global_step‚10849 % _timestamp‚1.7884154822368798e+09ú -Ê2œ'- _wandbruntime‚8092  _runtime‚8092µ¥;¾Z'Ô 0loss/FBt1/assignment_loss‚0.3269171233954694 1loss/FBt2/assignment_loss‚0.16066876437980682 /loss/FRt1/detection_loss‚2.8684587623133804 .loss/FRt2/detection_loss‚2.292924558970986 /loss/FBt4/detection_loss‚2.7635793328285216 1loss/SRqqt4/detection_loss‚1.7690788800162929 .loss/FBt1/detection_loss‚3.212894231081009 %loss/total_loss‚47.63943273256651 epoch‚8 /loss/FRt2/assignment_loss‚2.529215462324096 0loss/FRt4/assignment_loss‚3.1240087978693905 2loss/SRqqt4/assignment_loss‚1.4306818466367466 0loss/FBt4/assignment_loss‚0.4030199896772501 _runtime‚8092.651242313 _step‚891 .loss/FRt3/detection_loss‚1.956053180721673 .loss/FRt4/detection_loss‚2.140247864990818 0loss/SRqqt2/detection_loss‚1.511703839332893 trainer/global_step‚10849 0loss/FRt3/assignment_loss‚2.1798345468261027 1loss/SRqqt3/detection_loss‚1.5109967765613244 /loss/FBt3/detection_loss‚0.7069605041379103 /loss/FRt1/assignment_loss‚5.082284840670499 2loss/SRqqt1/assignment_loss‚2.4867915187416405 1loss/FBt3/assignment_loss‚0.18148708487358642 1loss/SRqqt1/detection_loss‚2.5165816350229857 /loss/FBt2/detection_loss‚3.3490297943353653 $ _timestamp‚1.788415482239178e+09 2loss/SRqqt2/assignment_loss‚1.3801765179197336 2loss/SRqqt3/assignment_loss‚1.7558368789390302*.­›_ž'Ù 2loss/SRqqt1/assignment_loss‚2.4867915187416405 1loss/FBt3/assignment_loss‚0.18148708487358642 1loss/SRqqt1/detection_loss‚2.5165816350229857 /loss/FBt2/detection_loss‚3.3490297943353653 $ _timestamp‚1.788415482239178e+09 2loss/SRqqt2/assignment_loss‚1.3801765179197336 2loss/SRqqt3/assignment_loss‚1.7558368789390302 0loss/FBt1/assignment_loss‚0.3269171233954694 1loss/FBt2/assignment_loss‚0.16066876437980682 /loss/FRt1/detection_loss‚2.8684587623133804 .loss/FRt2/detection_loss‚2.292924558970986 /loss/FBt4/detection_loss‚2.7635793328285216 1loss/SRqqt4/detection_loss‚1.7690788800162929 .loss/FBt1/detection_loss‚3.212894231081009 %loss/total_loss‚47.63943273256651 epoch‚8 /loss/FRt2/assignment_loss‚2.529215462324096 0loss/FRt4/assignment_loss‚3.1240087978693905 2loss/SRqqt4/assignment_loss‚1.4306818466367466 0loss/FBt4/assignment_loss‚0.4030199896772501 _runtime‚8092.651242313 _step‚891 .loss/FRt3/detection_loss‚1.956053180721673 .loss/FRt4/detection_loss‚2.140247864990818 0loss/SRqqt2/detection_loss‚1.511703839332893 trainer/global_step‚10849 0loss/FRt3/assignment_loss‚2.1798345468261027 1loss/SRqqt3/detection_loss‚1.5109967765613244 /loss/FBt3/detection_loss‚0.7069605041379103 /loss/FRt1/assignment_loss‚5.082284840670499û.˜‘Ÿ'‚(:† œäÔãÅ«š gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.398' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­º , '‚(:¡ œäÔþ¦õ› disk.sda.out‚ 322.8046875 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd3.in‚0) proc.memory.percent‚17.57500171661377 proc.cpu.threads‚17% disk.nvme0n1.out‚77017.2138671875 disk.rbd0.out‚0' disk.nvme0c0n1.out‚77017.2138671875) disk./.usagePercent‚73.59941170902273 disk.sda3.out‚ 322.8046875 disk.drbd1777.in‚0 disk.dm-0.in‚ 715.55859375 disk.dm-1.in‚0 network.sent‚ 2.6159601e+07 disk.sda3.in‚0 disk.rbd1.out‚ 4.35546875 disk.nvme0n1.in‚ 715.55859375 disk.rbd0.in‚0! disk.rbd2.out‚189.50634765625$ memory_percent‚98.48449230194092( proc.memory.availableMB‚ 248.30078125% disk./.usageGB‚1096.6844215393066 disk.sda2.in‚0 disk.sda.in‚0" disk.dm-0.out‚77017.2138671875 disk.dm-1.out‚ 322.8046875 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 715.55859375# proc.memory.rssMB‚ 2879.48828125 disk.sda1.in‚0 cpu‚25.361969496575032 disk.rbd3.out‚0 disk.drbd1777.out‚0 network.recv‚ 2.98855e+06 disk.sda2.out‚0»MøY¡'‚(:„ œäÔøÏ™ gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.48' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.896 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï¡ê©*¢'‚(:Ÿ œäÔߜÛ disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.35546875 disk.dm-0.in‚ 715.55859375 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚98.48299026489258& proc.memory.availableMB‚ 248.546875% disk./.usageGB‚1096.6844215393066 disk.dm-1.out‚ 323.140625 disk.rbd3.in‚0( disk.nvme0c0n1.out‚77017.53759765625 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 715.55859375) proc.memory.percent‚17.57500171661377 cpu‚25.36139712150333 proc.cpu.threads‚17 disk.sda3.out‚ 323.140625 disk.drbd1777.out‚0# proc.memory.rssMB‚ 2879.48828125& disk.nvme0n1.out‚77017.53759765625) disk./.usagePercent‚73.59941170902273 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.rbd2.out‚191.11962890625 disk.rbd3.out‚0 network.sent‚ 2.6267745e+07 network.recv‚ 2.995295e+06 disk.rbd2.in‚0 disk.sda.in‚0# disk.dm-0.out‚77017.53759765625" disk.nvme0c0n1.in‚ 715.55859375 disk.sda3.in‚0 disk.sda.out‚ 323.140625 disk.sda1.in‚0ûa@£'‚(:„ ŸœäÔ˜–±™ gpu.0.gpu‚7.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.219' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6438 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G ê.¤'‚(:£ ŸœäÔÝ×õœ% disk./.usageGB‚1096.6844215393066 disk.rbd1.out‚ 4.35546875" disk.dm-0.out‚77018.1748046875# proc.memory.rssMB‚ 2879.48828125 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 715.55859375$ memory_percent‚98.48299026489258 network.recv‚ 2.999027e+06 disk.sda.out‚ 324.04296875 disk.rbd1.in‚0 disk.dm-0.in‚ 715.55859375 disk.rbd2.in‚0! disk.rbd2.out‚191.11962890625 disk.sda2.out‚0% disk.nvme0n1.out‚77018.1748046875' disk.nvme0c0n1.out‚77018.1748046875 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd0.in‚0& proc.memory.availableMB‚ 248.546875 cpu‚25.369981928636136 disk.drbd1777.in‚0 disk.rbd3.in‚0) proc.memory.percent‚17.57500171661377) disk./.usagePercent‚73.59941170902273 disk.dm-1.out‚ 324.04296875 network.sent‚ 2.6273214e+07 disk.rbd0.out‚0 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 715.55859375 disk.sda3.out‚ 324.04296875‹áÜö¥'‚(:… ®œäÔ»†š™ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚84.429' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚16.8858 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Áï†F-¦'‚(:¢ ®œäÔ£¿æœ disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda.out‚ 324.33203125 disk.nvme0n1.in‚ 715.55859375$ memory_percent‚98.48299026489258 proc.cpu.threads‚17) disk./.usagePercent‚73.60779646778418 disk.dm-1.in‚0 disk.sda1.in‚0 network.sent‚ 2.6279459e+07! disk.rbd2.out‚192.86181640625 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 715.55859375 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1777.in‚0& proc.memory.availableMB‚ 248.546875 network.recv‚ 3.003852e+06 disk.dm-0.in‚ 715.55859375 disk.dm-1.out‚ 324.33203125 disk.rbd0.in‚0' disk.nvme0c0n1.out‚77117.1787109375 disk.sda.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚77117.1787109375# proc.memory.rssMB‚ 2879.48828125 disk.rbd3.out‚0% disk./.usageGB‚1096.8093605041504" disk.dm-0.out‚77117.1787109375) proc.memory.percent‚17.57500171661377 disk.sda3.out‚ 324.33203125 disk.sda1.out‚0 disk.rbd1.out‚ 4.35546875 cpu‚25.37851431589353»x>½§'‚(:„ ½œäÔüÒ£™ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êòÊ-¨'‚(:¢ ½œäÔžûל disk.sda1.in‚0$ memory_percent‚98.48449230194092 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.35546875 disk.nvme0n1.in‚ 715.55859375 network.recv‚ 3.008288e+06) disk./.usagePercent‚73.61614026534014 disk.sda2.in‚0 disk.dm-1.out‚ 324.88671875 disk.rbd0.out‚0& disk.nvme0c0n1.out‚77117.837890625 disk.sda3.out‚ 324.88671875 disk.sda.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 248.30078125) proc.memory.percent‚17.57500171661377 disk.rbd2.in‚0 disk.sda.out‚ 324.88671875 disk.dm-0.in‚ 715.55859375# proc.memory.rssMB‚ 2879.48828125 proc.cpu.threads‚17 disk.rbd3.in‚0$ disk.nvme0n1.out‚77117.837890625% disk./.usageGB‚1096.9336891174316 disk.dm-1.in‚0 disk.rbd0.in‚0! disk.rbd2.out‚192.86181640625 disk.sda2.out‚0 network.sent‚ 2.6285538e+07 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 715.55859375 disk.drbd1777.in‚0 cpu‚25.387229248611103 disk.sda3.in‚0! disk.dm-0.out‚77117.837890625W\Œ2©'- _wandbruntime‚8161  _runtime‚8161&mfæª'ÏT ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚15355793368064 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 145662516 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚341105633358336 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 5041553408 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 143687741 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 220153073 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 5041553408 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 4624220160 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚199 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 143687741 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 220153073 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚15311653634848 _runtime‚8161.865400484 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 4541568000 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚307190338007040 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚307190320004096 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚15355793368064 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 143687739 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚83879218 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚325482687372288 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚15623077498880 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 145311232 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚15622969926144 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 trainer/global_step‚10880 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 140964952 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 127308288 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 363840812 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 131512832 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 4757245257 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2460 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚83879220 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚217 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 107572736 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 _step‚892 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚322546258683392 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 46242201606ú9Z*«'ÔT ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚15355793368064 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 145662516 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚341105633358336 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 5041553408 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 143687741 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 220153073 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 5041553408 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 4624220160 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚199 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 143687741 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 220153073 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚15311653634848 _runtime‚8161.865400484 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 4541568000 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚307190338007040 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚307190320004096 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚15355793368064 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 143687739 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚83879218 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚325482687372288 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚15623077498880 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 145311232 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚15622969926144 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 trainer/global_step‚10880 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 140964952 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 127308288 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 363840812 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 131512832 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 4757245257 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2460 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚83879220 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚217 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 107572736 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 _step‚892 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚322546258683392 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2üðÒWe2¬'- _wandbruntime‚8162  _runtime‚8162†A(0^)­'ØR QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 145311232 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚83885720 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚341125717572608 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚307552908150792 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚307209185189888 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚217 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 363858920 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 107572736 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚15356269812224 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚15623319062016 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 5041553408 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 145311232 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚83885718 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 4912020992 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 143698453 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚61787724 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 131512832 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚15623426634752 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚199 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 4912020992 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚341125586059776 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 4757245257 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚8162.816599854 _step‚893 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 143698451 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 4541568000 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚15356142503936 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 410712064 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 417333248 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚217 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 127308288 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚307209203192832 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 410712064 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚15356269812224 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 406789969 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 220160469 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 145673160 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚292241030835200 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚15311877315592 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 143698453 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚322565473005056#LR„®'ÝR TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 145311232 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚83885718 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 4912020992 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 143698453 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚61787724 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 131512832 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚15623426634752 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚199 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 4912020992 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚341125586059776 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 4757245257 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚8162.816599854 _step‚893 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 143698451 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 4541568000 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚15356142503936 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 410712064 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 417333248 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚217 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 127308288 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚307209203192832 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 410712064 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚15356269812224 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 406789969 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 220160469 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 145673160 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚292241030835200 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚15311877315592 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 143698453 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚322565473005056 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 145311232 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚83885720 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚341125717572608 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚307552908150792 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚307209185189888 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚217 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 363858920 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 107572736 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚15356269812224 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚15623319062016 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 5041553408ýW´¢›¯'‚(: ÌœäÔëÉ¡œ gpu.0.gpu‚41.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚89.821' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.964199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²ÌÐh'°'‚(:œ ÌœäÔÀÜŸž disk.nvme0n1.in‚ 715.55859375$ disk./.usageGB‚1096.933666229248 disk.dm-1.in‚0 disk.dm-1.out‚ 325.32421875 disk.sda.out‚ 325.32421875 disk.drbd1777.in‚0 network.sent‚ 2.6290902e+07 network.recv‚ 3.012211e+06 disk.rbd0.in‚0! disk.rbd2.out‚194.53759765625 disk.rbd3.in‚0 disk.sda3.in‚0$ disk.nvme0n1.out‚77216.998046875 disk.sda2.in‚0 disk.sda1.out‚0$ memory_percent‚98.52786064147949' proc.memory.availableMB‚ 241.1953125) proc.memory.percent‚17.63019561767578& disk.nvme0c0n1.out‚77216.998046875 proc.cpu.threads‚17) disk./.usagePercent‚73.61613872929493 disk.dm-0.in‚ 715.55859375 disk.rbd0.out‚0 disk.sda.in‚0 disk.rbd1.out‚ 4.35546875 disk.sda3.out‚ 325.32421875 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.drbd1777.out‚0 proc.memory.rssMB‚ 2888.53125 cpu‚25.39611477718033! disk.dm-0.out‚77216.998046875 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 715.55859375 disk.sda2.out‚0à{Ez2±'- _wandbruntime‚8175  _runtime‚8175"Ö•¤ j²'Ǧ @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 particle/accuracy_4_of_5‚1 jet/accuracy_2_of_7‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.47450655698776245 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17803031206130981 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1812385767698288 +jet/accuracy_1_of_1‚0.47198018431663513 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1533561646938324 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.032894738018512726 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9870559573173523 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04434523731470108 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 +jet/accuracy_4_of_6‚0.01923076994717121 particle/accuracy_5_of_7‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 )jet/accuracy_2_of_4‚0.342836856842041 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.522988498210907 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.017924528568983078 7"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.52717524766922 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.022209644317626953 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.029069768264889717 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9851411581039429 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5028751492500305 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14368896186351776 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 jet/accuracy_3_of_7‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9649305939674377 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0019919080659747124 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1482379287481308 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_2‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08156028389930725 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5069444179534912 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10512619465589523 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5087466835975647 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.26203709840774536 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.484883189201355 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.5512820482254028 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 _step‚894 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6790165901184082 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2207791656255722 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3714558780193329 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.4065934419631958 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 jet/accuracy_5_of_5‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4660581648349762 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.38876423239707947 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1673334836959839 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9952027797698975 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 particle/accuracy_1_of_3‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3756624460220337 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3517361581325531 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1381712406873703 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2430555522441864 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.47013920545578003 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4041890501976013 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13618281483650208 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5474014282226562 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9818713665008545 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2033480405807495 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4686220586299896 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5387868881225586 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5116309523582458 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9912132024765015 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5350434184074402 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15198683738708496 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.15856872498989105 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1387687623500824 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5140153765678406 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.11973179876804352 *jet/accuracy_1_of_4‚0.8978327512741089 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4791666567325592 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23567739129066467 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_6‚0.9615384340286255 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4450698792934418 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1964285671710968 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_6_of_7‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05902700498700142 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 particle/accuracy_3_of_3‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5443704724311829 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5010547041893005 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 +"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.25 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02957495115697384 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.988676130771637 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14580358564853668 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 8"Purity/1FRt*SRqqt0FBt/event_purity‚0.068830706179142 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9696621298789978 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9908046126365662 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4953652024269104 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 epoch‚8 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07576102763414383 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4548611640930176 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17464250326156616 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.46020761132240295 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.021995464339852333 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3682170510292053 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5298231244087219 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5100694298744202 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010257013142108917 ,jet/accuracy_4_of_5‚0.004480286967009306 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9698275327682495 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5494332909584045 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3166666626930237 particle/accuracy_2_of_6‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9893742799758911 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.039819374680519104 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24022988975048065 particle/accuracy_1_of_1‚1 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_7‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4199416935443878 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9873238205909729 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_6‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4826142191886902 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18694280087947845 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23355567455291748 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4562733471393585 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.015205341391265392 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.40120118856430054 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5779765248298645 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0417913943529129 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0912698432803154 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5364326238632202 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.31323596835136414 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.48579978942871094 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43827158212661743 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.35838204622268677 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11831274628639221 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.375 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3703703284263611 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14580358564853668 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.002765753073617816 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29029157757759094 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23720569908618927 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.25649896264076233 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16204263269901276 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5418498516082764 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.37026581168174744 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5147078037261963 (particle/f_score‚0.39247286319732666 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1824733167886734 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.142875075340271 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5092592835426331 +jet/accuracy_3_of_4‚0.05599229037761688 )jet/accuracy_3_of_5‚0.140962615609169 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9711053371429443 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4973357021808624 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9811145067214966 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5912697911262512 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 )jet/accuracy_1_of_5‚0.940693736076355 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4461021423339844 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.018298771232366562 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.33529454469680786 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10467108339071274 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.32760709524154663 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9910348653793335 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17038442194461823 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5789473652839661 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5117630362510681 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4474981427192688 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5538468956947327 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ,jet/accuracy_3_of_6‚0.057692307978868484 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.36666667461395264 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009542706422507763 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.44065219163894653 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4252173900604248 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.05263157933950424 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365078806877136 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9841199517250061 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3718954026699066 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9963939189910889 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9923290610313416 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11223858594894409 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4765802025794983 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9897217750549316 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.19367815554141998 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6818181872367859 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.0625 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_2_of_7‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3725505769252777 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.30000001192092896 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21306806802749634 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18043826520442963 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4999999701976776 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.300548255443573 particle/accuracy_2_of_3‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5400000214576721 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.017153147608041763 +jet/accuracy_2_of_3‚0.23153440654277802 particle/accuracy_1_of_2‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3196428418159485 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5333333611488342 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2148856818675995 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 particle/accuracy_4_of_4‚1 +particle/sensitivity‚0.3094378709793091 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.36982569098472595 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.47345224022865295 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.36443451046943665 )jet/accuracy_1_of_2‚0.737170398235321 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184397459030151 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0“â­D) 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9936589002609253 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_6_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 particle/accuracy_6_of_7‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13531549274921417 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4346388280391693 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 particle/accuracy_3_of_5‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.09777776896953583 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9885329008102417 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5214611291885376 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6818181872367859 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 +jet/accuracy_2_of_2‚0.10969149321317673 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3793173134326935 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.022379819303750992 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4963876008987427 particle/accuracy_1_of_5‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 particle/accuracy_7_of_7‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 )loss/val_total_loss‚47.56908641648854 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06770152598619461 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.24292656779289246 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1341332048177719 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.17004504799842834 particle/accuracy_2_of_5‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1240934506058693 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16617128252983093 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.026860445737838745 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13072778284549713 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5364367365837097 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9859274625778198 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3934343457221985 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 *particle/specificity‚0.950650691986084 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151411294937 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.44733330607414246 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.45660918951034546 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19737817347049713 *jet/accuracy_1_of_3‚0.8488177061080933 ,jet/accuracy_4_of_4‚0.002213064581155777 jet/accuracy_5_of_6‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.39626696705818176 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5128011703491211 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 jet/accuracy_7_of_7‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2007792741060257 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 trainer/global_step‚10880 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9823309183120728 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08644436299800873 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3242107033729553 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3969298303127289 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4847285747528076 jet/accuracy_5_of_7‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.484883189201355 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13524360954761505 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.35227271914482117 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 _runtime‚8175.512930712 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9814563989639282 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.36732468008995056 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.29321879148483276 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.18771930038928986 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9855158925056458 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13717246055603027 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24220657348632812 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15228618681430817 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6403061151504517 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.17515434324741364 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02577066980302334 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5273376703262329 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_6‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.997589111328125 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5354166626930237 jet/accuracy_4_of_7‚0 % _timestamp‚1.7884155650987725e+09 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4791666567325592 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.36344802379608154 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8294169902801514 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005219993181526661 (particle/accuracy‚0.8505446314811707 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.45294204354286194 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 *"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5628072023391724 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0¶†P7®V³'̦ 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5140153765678406 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.11973179876804352 *jet/accuracy_1_of_4‚0.8978327512741089 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4791666567325592 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23567739129066467 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_6‚0.9615384340286255 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4450698792934418 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1964285671710968 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_6_of_7‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05902700498700142 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 particle/accuracy_3_of_3‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5443704724311829 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5010547041893005 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 +"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.25 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02957495115697384 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.988676130771637 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14580358564853668 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 8"Purity/1FRt*SRqqt0FBt/event_purity‚0.068830706179142 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9696621298789978 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9908046126365662 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4953652024269104 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 epoch‚8 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07576102763414383 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4548611640930176 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17464250326156616 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.46020761132240295 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.021995464339852333 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3682170510292053 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5298231244087219 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5100694298744202 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010257013142108917 ,jet/accuracy_4_of_5‚0.004480286967009306 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9698275327682495 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5494332909584045 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3166666626930237 particle/accuracy_2_of_6‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9893742799758911 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.039819374680519104 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24022988975048065 particle/accuracy_1_of_1‚1 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_7‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4199416935443878 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9873238205909729 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_6‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4826142191886902 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18694280087947845 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23355567455291748 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4562733471393585 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.015205341391265392 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.40120118856430054 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5779765248298645 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0417913943529129 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0912698432803154 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5364326238632202 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.31323596835136414 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.48579978942871094 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43827158212661743 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.35838204622268677 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11831274628639221 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.375 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3703703284263611 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14580358564853668 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.002765753073617816 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29029157757759094 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23720569908618927 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.25649896264076233 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16204263269901276 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5418498516082764 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.37026581168174744 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5147078037261963 (particle/f_score‚0.39247286319732666 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1824733167886734 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.142875075340271 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5092592835426331 +jet/accuracy_3_of_4‚0.05599229037761688 )jet/accuracy_3_of_5‚0.140962615609169 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9711053371429443 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4973357021808624 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9811145067214966 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5912697911262512 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 )jet/accuracy_1_of_5‚0.940693736076355 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4461021423339844 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.018298771232366562 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.33529454469680786 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10467108339071274 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.32760709524154663 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9910348653793335 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17038442194461823 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5789473652839661 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5117630362510681 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4474981427192688 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5538468956947327 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ,jet/accuracy_3_of_6‚0.057692307978868484 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.36666667461395264 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009542706422507763 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.44065219163894653 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4252173900604248 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.05263157933950424 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365078806877136 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9841199517250061 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3718954026699066 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9963939189910889 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9923290610313416 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11223858594894409 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4765802025794983 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9897217750549316 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.19367815554141998 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6818181872367859 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.0625 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_2_of_7‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3725505769252777 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.30000001192092896 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21306806802749634 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18043826520442963 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4999999701976776 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.300548255443573 particle/accuracy_2_of_3‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5400000214576721 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.017153147608041763 +jet/accuracy_2_of_3‚0.23153440654277802 particle/accuracy_1_of_2‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3196428418159485 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5333333611488342 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.2148856818675995 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 particle/accuracy_4_of_4‚1 +particle/sensitivity‚0.3094378709793091 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.36982569098472595 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.47345224022865295 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.36443451046943665 )jet/accuracy_1_of_2‚0.737170398235321 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184397459030151 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9936589002609253 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_6_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 particle/accuracy_6_of_7‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13531549274921417 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4346388280391693 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 particle/accuracy_3_of_5‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.09777776896953583 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9885329008102417 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5214611291885376 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6818181872367859 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 +jet/accuracy_2_of_2‚0.10969149321317673 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3793173134326935 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.022379819303750992 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4963876008987427 particle/accuracy_1_of_5‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 particle/accuracy_7_of_7‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 )loss/val_total_loss‚47.56908641648854 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06770152598619461 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.24292656779289246 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1341332048177719 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.17004504799842834 particle/accuracy_2_of_5‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1240934506058693 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16617128252983093 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ;"Purity/2FRt0SRqÒ~Yu¥<qt*FBt/event_purity‚0.026860445737838745 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13072778284549713 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5364367365837097 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9859274625778198 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3934343457221985 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 *particle/specificity‚0.950650691986084 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151411294937 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.44733330607414246 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.45660918951034546 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19737817347049713 *jet/accuracy_1_of_3‚0.8488177061080933 ,jet/accuracy_4_of_4‚0.002213064581155777 jet/accuracy_5_of_6‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.39626696705818176 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5128011703491211 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 jet/accuracy_7_of_7‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2007792741060257 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 trainer/global_step‚10880 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9823309183120728 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08644436299800873 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3242107033729553 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3969298303127289 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4847285747528076 jet/accuracy_5_of_7‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.484883189201355 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13524360954761505 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.35227271914482117 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 _runtime‚8175.512930712 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9814563989639282 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.36732468008995056 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.29321879148483276 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.18771930038928986 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9855158925056458 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13717246055603027 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24220657348632812 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15228618681430817 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6403061151504517 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.17515434324741364 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02577066980302334 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5273376703262329 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_6‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.997589111328125 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5354166626930237 jet/accuracy_4_of_7‚0 % _timestamp‚1.7884155650987725e+09 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4791666567325592 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.36344802379608154 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8294169902801514 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005219993181526661 (particle/accuracy‚0.8505446314811707 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.45294204354286194 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 *"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5628072023391724 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 particle/accuracy_4_of_5‚1 jet/accuracy_2_of_7‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.47450655698776245 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17803031206130981 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1812385767698288 +jet/accuracy_1_of_1‚0.47198018431663513 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1533561646938324 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.032894738018512726 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9870559573173523 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04434523731470108 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 +jet/accuracy_4_of_6‚0.01923076994717121 particle/accuracy_5_of_7‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 )jet/accuracy_2_of_4‚0.342836856842041 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.522988498210907 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.017924528568983078 7"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.52717524766922 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.022209644317626953 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.029069768264889717 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9851411581039429 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5028751492500305 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14368896186351776 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 jet/accuracy_3_of_7‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9649305939674377 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0019919080659747124 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1482379287481308 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_2‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08156028389930725 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5069444179534912 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10512619465589523 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5087466835975647 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.26203709840774536 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.484883189201355 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.5512820482254028 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 _step‚894 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6790165901184082 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2207791656255722 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3714558780193329 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.4065934419631958 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 jet/accuracy_5_of_5‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4660581648349762 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.38876423239707947 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1673334836959839 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9952027797698975 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 particle/accuracy_1_of_3‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3756624460220337 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3517361581325531 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1381712406873703 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2430555522441864 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.47013920545578003 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4041890501976013 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13618281483650208 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5474014282226562 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9818713665008545 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2033480405807495 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4686220586299896 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5387868881225586 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5116309523582458 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9912132024765015 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5350434184074402 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15198683738708496 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.15856872498989105 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1387687623500824þ…þ¦/›´'‚(: ÛœäÔÿ²™ gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.751' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.750200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\-6#µ'‚(:˜ ÛœäÔî’Öœ network.recv‚ 3.018888e+06 disk.rbd2.in‚0 disk.sda3.in‚0 disk.drbd1777.out‚0! disk.dm-0.out‚77439.341796875" disk.nvme0c0n1.in‚ 715.55859375& disk.nvme0c0n1.out‚77439.341796875 disk.nvme0n1.in‚ 715.55859375$ disk.nvme0n1.out‚77439.341796875 disk.dm-1.in‚0 disk.sda.in‚0 disk.rbd1.out‚ 4.35546875 disk.dm-0.in‚ 715.55859375( proc.memory.availableMB‚ 376.05859375 disk.dm-1.out‚ 325.890625 disk.rbd3.in‚0 disk.sda1.in‚0 cpu‚25.39552346641213 network.sent‚ 2.6399157e+07) disk./.usagePercent‚73.63295125608356 disk.rbd3.out‚0 disk.sda2.in‚0 disk.drbd1777.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1097.1841850280762 disk.rbd0.out‚0 disk.sda.out‚ 325.890625$ memory_percent‚97.70472049713135 proc.memory.rssMB‚ 2888.53125) proc.memory.percent‚17.63019561767578 disk.rbd0.in‚0! disk.rbd2.out‚194.53759765625 disk.sda2.out‚0 disk.sda3.out‚ 325.890625 disk.sda1.out‚0 disk.rbd1.in‚0µ¿ƒv2¶'- _wandbruntime‚8190  _runtime‚8190Ü—øÏÈ·' trainer/global_step‚10899 % _timestamp‚1.7884155802082932e+09 _runtime‚8190.620470534 _step‚895 & lr-AdamW/pg1‚2.957418963182257e-06 & lr-AdamW/pg2‚2.957418963182257e-06˜Yï͸'Ç _runtime‚8190.620470534 _step‚895 & lr-AdamW/pg1‚2.957418963182257e-06 & lr-AdamW/pg2‚2.957418963182257e-06 trainer/global_step‚10899 % _timestamp‚1.7884155802082932e+09ÿPÍ2¹'- _wandbruntime‚8190  _runtime‚8190Q=Þî'º'èO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚307945281170432 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15385895520768 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚292940728314880 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 220613270 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15385895520768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚308282096986914 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323331007691264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚307945299173376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚896 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 trainer/global_step‚10899 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚323331194694144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚323331194694144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 _runtime‚8190.621801251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚326277570358272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 220633308 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61936386 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚326277546418176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚307945299173376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 364678839 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61936875 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚489 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884155802092457e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146039390 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚341931189711360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15341368672034 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144065569 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15385726520832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15653406671360 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 220633308 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚323331007691264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚491 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84103006 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144065571 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚307945299173376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 364678839 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61936875 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚489 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884155802092457e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146039390 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚341931189711360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15341368672034 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144065569 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15385726520832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15653406671360 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 220633308 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚323331007691264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚491 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84103006 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144065571 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323331007691264€Å2¼'- _wandbruntime‚8191  _runtime‚8191o€y÷&½'ñM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 364731351 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144078228 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 220653121 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚307972568538112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84110381 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146051935 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚326306459366400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 146052437 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚308309429333794 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61942054 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚341961286441984 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15655033452544 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15342922958190 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15387288672256 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚341961516759040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚292966671497216 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15387288672256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _step‚897 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 220633057 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 364711285 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚502 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚326306483306496 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144078230 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚500 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 220653121²†¡ ü&¾'öM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 $ _timestamp‚1.788415580881593e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144078228 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15387119659008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84110383 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚307972568538112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84110381 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146051935 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚326306459366400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 146052437 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚308309429333794 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61942054 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚341961286441984 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15655033452544 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15342922958190 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15387288672256 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚341961516759040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚292966671497216 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15387288672256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _step‚897 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 220633057 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 364711285 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚502 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚326306483306496 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144078230 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚500 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 220653121 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 364731351 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144078228 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 220653121 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚324764147250688 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚309648727922122 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚294237887431680 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚324764147250688 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 146677016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 366304491 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚309308767410176 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15455566843392 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚343447641178624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚327723193829376 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15455566843392 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚523 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 % _timestamp‚1.7884156154240859e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62204639 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15724210727424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚309308749407232 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15455397843456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 221605972 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚309308767410176 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15724423409152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 366324531 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144698519 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146676493 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚327723217769472 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144698521 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 221626010 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62205160 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ø]Úó'Ë'íO _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15455566843392 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚343447641178624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚327723193829376 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15455566843392 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚523 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 % _timestamp‚1.7884156154240859e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62204639 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15724210727424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚309308749407232 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15455397843456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 221605972 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚309308767410176 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15724423409152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 366324531 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144698519 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146676493 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚327723217769472 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144698521 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 221626010 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62205160 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚324764334253568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚294237904471040 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 221626010 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15410840490442 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 _step‚900 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚324764334253568 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15455397843456 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144698519 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84471856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚8225.836601962 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144698521 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 221605972 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚309308749407232 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚343447404556800 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚309648910082886 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚324764147250688 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚309648727922122 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚294237887431680 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚324764147250688 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 146677016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 366304491 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚309308767410176„šê„2Ì'- _wandbruntime‚8226  _runtime‚8226@€È%÷&Í'ñM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚309676242429974 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15725626838016 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚324792996769792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚327752106777600 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84479231 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚309336018771968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15456790981632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 366336937 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 366336937 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144711180 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚324792996769792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚343477733615616 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚294263830614016 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚309676060269002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _step‚901 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62210433 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 % _timestamp‚1.7884156160971704e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15412229654986 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 _runtime‚ 8226.51010509 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146689135 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 366357003 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 366357003 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚294263847653376 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 221625759 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚10949 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15456959994880 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 221645823 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2@hód Î'öM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚324792996769792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚327752106777600 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84479231 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚309336018771968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15456790981632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 366336937 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 366336937 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144711180 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚324792996769792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚343477733615616 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚294263830614016 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚309676060269002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _step‚901 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62210433 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 % _timestamp‚1.7884156160971704e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15412229654986 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 Pˆw˜tor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 146689666 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144711178 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15725835312128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 _runtime‚ 8226.51010509 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146689135 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 366357003 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 366357003 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚294263847653376 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 221625759 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚10949 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15456959994880 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 221645823 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚309676242429974 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15725626838016 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199…î‡Î2Ï'- _wandbruntime‚8226  _runtime‚8226ñðmYXÐ'Ò /loss/FBt1/detection_loss‚2.6358813775910273 /loss/FBt3/detection_loss‚0.8482523349420805 /loss/FRt3/assignment_loss‚2.607120328148206 2loss/SRqqt3/assignment_loss‚1.4491615599747432 0loss/FBt2/assignment_loss‚0.8130819797515869 0loss/SRqqt1/detection_loss‚2.574958418806394 /loss/FBt4/detection_loss‚2.6840539243486194 /loss/FRt2/assignment_loss‚2.671332581246153 2loss/SRqqt4/assignment_loss‚1.3912904366540413 0loss/SRqqt2/detection_loss‚1.659404024271898 % _timestamp‚1.7884156160991595e+09 0loss/FRt1/assignment_loss‚4.7035056215066176 /loss/FRt3/detection_loss‚1.8170583409567673 %loss/total_loss‚52.69295622999485 1loss/SRqqt1/assignment_loss‚2.139088864955637 1loss/FBt4/assignment_loss‚0.48495672974927906 .loss/FRt1/detection_loss‚2.928014237147111 /loss/FRt2/detection_loss‚2.1419336206101356 /loss/FRt4/detection_loss‚2.1470878693792557 1loss/SRqqt3/detection_loss‚1.5225697806326008 epoch‚9 0loss/FBt1/assignment_loss‚0.4475370057300299 1loss/SRqqt4/detection_loss‚1.9446533753847082 .loss/FBt2/detection_loss‚8.629318356513977 trainer/global_step‚10949 _runtime‚ 8226.51124088 _step‚902 /loss/FRt4/assignment_loss‚3.225523544682397 1loss/SRqqt2/assignment_loss‚1.032108812317462 1loss/FBt3/assignment_loss‚0.19506310469412697ûŽ)@]Ñ'× /loss/FRt2/assignment_loss‚2.671332581246153 2loss/SRqqt4/assignment_loss‚1.3912904366540413 0loss/SRqqt2/detection_loss‚1.659404024271898 % _timestamp‚1.7884156160991595e+09 0loss/FRt1/assignment_loss‚4.7035056215066176 /loss/FRt3/detection_loss‚1.8170583409567673 %loss/total_loss‚52.69295622999485 1loss/SRqqt1/assignment_loss‚2.139088864955637 1loss/FBt4/assignment_loss‚0.48495672974927906 .loss/FRt1/detection_loss‚2.928014237147111 /loss/FRt2/detection_loss‚2.1419336206101356 /loss/FRt4/detection_loss‚2.1470878693792557 1loss/SRqqt3/detection_loss‚1.5225697806326008 epoch‚9 0loss/FBt1/assignment_loss‚0.4475370057300299 1loss/SRqqt4/detection_loss‚1.9446533753847082 .loss/FBt2/detection_loss‚8.629318356513977 trainer/global_step‚10949 _runtime‚ 8226.51124088 _step‚902 /loss/FRt4/assignment_loss‚3.225523544682397 1loss/SRqqt2/assignment_loss‚1.032108812317462 1loss/FBt3/assignment_loss‚0.19506310469412697 /loss/FBt1/detection_loss‚2.6358813775910273 /loss/FBt3/detection_loss‚0.8482523349420805 /loss/FRt3/assignment_loss‚2.607120328148206 2loss/SRqqt3/assignment_loss‚1.4491615599747432 0loss/FBt2/assignment_loss‚0.8130819797515869 0loss/SRqqt1/detection_loss‚2.574958418806394 /loss/FBt4/detection_loss‚2.6840539243486194†ÃŸ9›Ò'‚(: ˆäÔàô¶™ gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.993' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.798599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n·V-"Ó'‚(:— ˆäÔÛ»è› disk.rbd1.in‚0 proc.memory.rssMB‚ 2888.53125 disk.rbd1.out‚ 4.35546875 disk.nvme0n1.in‚ 715.55859375 disk.rbd0.in‚0 disk.sda.out‚ 327.4296875 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 715.55859375 disk.dm-1.in‚0$ memory_percent‚97.70369529724121 network.sent‚ 2.6636472e+07% disk./.usageGB‚1097.1839141845703 disk.sda1.out‚0 proc.cpu.threads‚17 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 715.55859375 disk.sda.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚73.63293307954862 disk.sda2.in‚0# disk.nvme0n1.out‚77761.44140625 disk.dm-0.out‚77761.44140625) proc.memory.percent‚17.63019561767578 disk.sda2.out‚0 disk.sda3.in‚0' proc.memory.availableMB‚ 376.2265625 cpu‚25.39358586484375 network.recv‚ 3.036996e+06 disk.sda3.out‚ 327.4296875 disk.dm-1.out‚ 327.4296875 disk.rbd2.in‚0! disk.rbd2.out‚197.83837890625 disk.rbd3.in‚0% disk.nvme0c0n1.out‚77761.44140625 disk.drbd1777.in‚0ý W¸Ô'‚(:… —äÔÚÊš™ gpu.0.gpu‚25.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.786' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.1572 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝÓÕ'‚(: —äÔÊìÄœ( proc.memory.availableMB‚ 375.98046875) proc.memory.percent‚17.63019561767578 network.sent‚ 2.674454e+07 network.recv‚ 3.043601e+06( disk./.usagePercent‚73.6664872190389 disk.sda2.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.dm-1.out‚327.875 disk.rbd0.out‚0 disk.sda3.out‚327.875 disk.rbd1.in‚0! disk.rbd2.out‚199.35009765625" disk.nvme0c0n1.in‚ 715.55859375 disk.rbd1.out‚ 4.35546875% disk.nvme0n1.out‚77860.8115234375 cpu‚25.39298869773457 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda.out‚327.875" disk.dm-0.out‚77860.8115234375$ memory_percent‚97.70519733428955 disk.rbd2.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 715.55859375$ disk./.usageGB‚1097.683895111084 disk.rbd0.in‚0 disk.rbd3.out‚0 proc.memory.rssMB‚ 2888.53125' disk.nvme0c0n1.out‚77860.8115234375 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 715.55859375ÁúRc2Ö'- _wandbruntime‚8260  _runtime‚8260ôÇb³Ê×'Ä trainer/global_step‚10999 % _timestamp‚1.7884156505679724e+09 _runtime‚8260.980186356 _step‚903 ' lr-AdamW/pg1‚2.4855845786166397e-06 ' lr-AdamW/pg2‚2.4855845786166397e-06=.G­ÏØ'É _runtime‚8260.980186356 _step‚903 ' lr-AdamW/pg1‚2.4855845786166397e-06 ' lr-AdamW/pg2‚2.4855845786166397e-06 trainer/global_step‚10999 % _timestamp‚1.7884156505679724e+09‡$VßØ2Ù'- _wandbruntime‚8260  _runtime‚8260‹ŸJéî'Ú'èO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚310672217644032 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚344963961316352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15525238166016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15795120075776 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 367930143 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15480477430254 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 222598674 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15795334854656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚8260.981096752 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145331471 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 trainer/global_step‚10999 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62475162 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 % _timestamp‚1.7884156505687392e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚326197473812992 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15525069166080 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145331469 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84840706 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚329168841240576 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145331469 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚326197286810112 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 147316386 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84840704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15525069166080 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15480312308850 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚518 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145331471 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15525238166016 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚310672235646976 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 222618712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 trainer/global_step‚10999 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62475162 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 % _timestamp‚1.7884156505687392e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚326197473812992 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15525069166080 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145331469 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84840706 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚329168841240576 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145331469 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚326197286810112 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 147316386 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84840704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15525069166080 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15480312308850 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚518 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145331471 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15525238166016 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚310672235646976 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 222618712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚311042691204210 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15526631317504 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚310699505011712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15526462304256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62480972 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚344994529532416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 222638525 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚344994286632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15481866595006 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15481701473394 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147328532 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚329197778128896 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15526462304256 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145344130 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84848083 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚295561006770176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚326226136329216 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚311042873365182 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 367962589 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚310699487008768 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84848081 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 367982655 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145344128 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 $ _timestamp‚1.788415651226104e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 222638525 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 222618461 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚329197754188800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62480451 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145344128 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 367982655 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 _runtime‚8261.638792631 _step‚905 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192S3Aü&Þ'öM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15796751403520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 222618461 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚326226136329216 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15526631317504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15796532443648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147329055 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 $ _timestamp‚1.788415651226104e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 222638525 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 222618461 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚329197754188800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62480451 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145344128 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 367982655 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 _runtime‚8261.638792631 _step‚905 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 367962589 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚311042691204210 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15526631317504 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚310699505011712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15526462304256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62480972 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚344994529532416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 222638525 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚344994286632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15481866595006 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15481701473394 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147328532 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚329197778128896 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15526462304256 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145344130 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84848083 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚295561006770176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚326226136329216 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚311042873365182 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 367962589 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚310699487008768 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84848081 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 367982655 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145344128 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 Fè"à*å'‚(:Ÿ µäÔþ‡œ) disk./.usagePercent‚73.66648747504644 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚77961.04150390625" disk.nvme0c0n1.in‚ 715.55859375 disk.sda2.in‚0$ memory_percent‚97.71420955657959 proc.cpu.threads‚17 network.sent‚ 2.6859646e+07 disk.rbd1.in‚0# disk.dm-0.out‚77961.04150390625 disk.rbd2.in‚0! disk.rbd2.out‚201.04541015625 proc.memory.rssMB‚ 2888.53125 network.recv‚ 3.055934e+06( disk.nvme0c0n1.out‚77961.04150390625 disk.drbd1777.out‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 715.55859375 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 328.828125) proc.memory.percent‚17.63019561767578 cpu‚25.391722530428428% disk./.usageGB‚1097.6838989257812( proc.memory.availableMB‚ 374.50390625 disk.rbd1.out‚ 4.35546875 disk.nvme0n1.in‚ 715.55859375 disk.dm-1.in‚0 disk.dm-1.out‚ 328.828125 disk.rbd3.in‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 328.828125 disk.rbd0.in‚0 disk.rbd3.out‚0ÂÙ8Œšæ'‚(: ÄäÔň¿™ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.648' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.729599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚDwR-ç'‚(:¢ ÄäÔø™„œ' disk.nvme0c0n1.out‚78068.0537109375 disk.sda2.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.rbd0.in‚0 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 715.55859375 disk.sda3.in‚0 disk.sda.in‚0 network.recv‚ 3.059893e+06) disk./.usagePercent‚73.66698796977587 disk.rbd3.out‚0 disk.sda2.out‚0 disk.sda.out‚ 329.30859375" disk.nvme0c0n1.in‚ 715.55859375% disk.nvme0n1.out‚78068.0537109375 disk.dm-1.in‚0% disk./.usageGB‚1097.6913566589355 disk.sda3.out‚ 329.30859375! disk.rbd2.out‚201.14697265625 proc.memory.rssMB‚ 2888.53125 cpu‚25.391106613374347 disk.rbd1.in‚0 disk.rbd1.out‚ 4.35546875( proc.memory.availableMB‚ 250.43359375) proc.memory.percent‚17.63019561767578 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 715.55859375" disk.dm-0.out‚78068.0537109375 disk.dm-1.out‚ 329.30859375 disk.rbd0.out‚0 disk.rbd2.in‚0$ memory_percent‚98.47147464752197 network.sent‚ 2.6865067e+07ð°ûƒ2è'- _wandbruntime‚8296  _runtime‚8296ú“ÛÊé'Ä ' lr-AdamW/pg1‚2.2646579565595983e-06 ' lr-AdamW/pg2‚2.2646579565595983e-06 trainer/global_step‚11049 % _timestamp‚1.7884156858821502e+09 _runtime‚8296.294334858 _step‚907HaÏê'É ' lr-AdamW/pg1‚2.2646579565595983e-06 ' lr-AdamW/pg2‚2.2646579565595983e-06 trainer/global_step‚11049 % _timestamp‚1.7884156858821502e+09 _runtime‚8296.294334858 _step‚907‹„ÇbÉ2ë'- _wandbruntime‚8296  _runtime‚8296«ø ì'èO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚312035685880832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62746080 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15549784127258 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85209554 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15594740488704 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 223591376 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚312381989792538 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚312382171953302 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚327630426369536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚330614488651776 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚346480492466688 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15549949248662 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 % _timestamp‚1.7884156858828857e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚908 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 147955636 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15594909488640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚327630426369536 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 _runtime‚8296.295255743 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚327630613372416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15594740488704 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15866003814912 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚31:`†“â2035685880832 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15866212302336 trainer/global_step‚11049 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 223591376 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145964419 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 369555795 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15594909488640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 223611414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85209556 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚312035703883776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 369575835 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚296832205665280YMæ ó'í'íO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15594909488640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚327630426369536 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 _runtime‚8296.295255743 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚327630613372416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15594740488704 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15866003814912 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚312035685880832 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15866212302336 trainer/global_step‚11049 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 223591376 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145964419 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 369555795 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15594909488640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 223611414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85209556 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚312035703883776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 369575835 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚296832205665280 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚312035685880832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62746080 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15549784127258 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85209554 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15594740488704 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 223591376 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚312381989792538 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚312382171953302 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚327630426369536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚330614488651776 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚346480492466688 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15549949248662 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 % _timestamp‚1.7884156858828857e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚908 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 147955636 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160Œ^É2î'- _wandbruntime‚8296  _runtime‚8296Ùé¢Ëø&ï'òM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚327659275888640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62750865 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 369608307 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15596302640128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚330643425540096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 223631227 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚327659275888640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 369588241 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85216931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚312062973248512 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15551173291802 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15596133626880 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15867423195136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15867631669248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚312062973248512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚346511057209344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85216933 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 369608307 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15551338413414 trainer/global_step‚11049 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145977080 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 369588241 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 223611163 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚330643401600000 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147967796 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚346510824795136 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚312409322139418 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145977078 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145977078 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚296858165886976 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚296858148847616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 223611163 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 _step‚909[*)æÞð'÷M J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15596302640128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚330643425540096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 223631227 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚327659275888640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 369588241 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85216931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚312062973248512 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15551173291802 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15596133626880 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15867423195136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15867631669248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚312062973248512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚346511057209344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85216933 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 369608307 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15551338413414 trainer/global_step‚11049 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145977080 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 369588241 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 223611163 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚330643401600000 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147967796 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚346510824795136 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚312409322139418 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145977078 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145977078 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚296858165886976 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚296858148847616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 223611163 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 _step‚909 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚327659275888640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62750865 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 369608307€.û52ñ'- _wandbruntime‚8296  _runtime‚8296—™aVò'Ð 0loss/FRt3/assignment_loss‚2.0569462102392446 .loss/FRt3/detection_loss‚2.355268218918987 /loss/FBt2/detection_loss‚2.3142039246029324 &loss/total_loss‚48.050145580739546 trainer/global_step‚11049 2loss/SRqqt1/assignment_loss‚2.5544958412647247 .loss/FBt4/detection_loss‚3.408919471502304 0loss/SRqqt3/detection_loss‚1.287006534084126 1loss/SRqqt2/assignment_loss‚1.328463251091307 /loss/FBt2/assignment_loss‚0.678987741884258 1loss/SRqqt1/detection_loss‚2.5307811349630356 _runtime‚ 8296.94765255 _step‚910 .loss/FRt1/assignment_loss‚4.64891619682312 /loss/FRt4/detection_loss‚2.1645608596942005 0loss/SRqqt4/detection_loss‚1.717699982225895 % _timestamp‚1.7884156865352483e+09 1loss/SRqqt2/detection_loss‚1.7204396254383028 /loss/FBt1/detection_loss‚2.8775843262672423 /loss/FBt3/detection_loss‚0.7542007037498626 epoch‚9 /loss/FRt4/assignment_loss‚3.280070303701887 /loss/FBt1/assignment_loss‚0.443912449117488 /loss/FRt2/assignment_loss‚2.618666888551509 2loss/SRqqt4/assignment_loss‚1.7142888474544244 1loss/FBt3/assignment_loss‚0.26161314711071465 0loss/FBt4/assignment_loss‚0.5389816432402768 /loss/FRt1/detection_loss‚3.2273761558532716 2loss/SRqqt3/assignment_loss‚1.3117707035055868 /loss/FRt2/detection_loss‚2.2549914194548384¦ø-h[ó'Õ /loss/FRt4/assignment_loss‚3.280070303701887 /loss/FBt1/assignment_loss‚0.443912449117488 /loss/FRt2/assignment_loss‚2.618666888551509 2loss/SRqqt4/assignment_loss‚1.7142888474544244 1loss/FBt3/assignment_loss‚0.26161314711071465 0loss/FBt4/assignment_loss‚0.5389816432402768 /loss/FRt1/detection_loss‚3.2273761558532716 2loss/SRqqt3/assignment_loss‚1.3117707035055868 /loss/FRt2/detection_loss‚2.2549914194548384 0loss/FRt3/assignment_loss‚2.0569462102392446 .loss/FRt3/detection_loss‚2.355268218918987 /loss/FBt2/detection_loss‚2.3142039246029324 &loss/total_loss‚48.050145580739546 trainer/global_step‚11049 2loss/SRqqt1/assignment_loss‚2.5544958412647247 .loss/FBt4/detection_loss‚3.408919471502304 0loss/SRqqt3/detection_loss‚1.287006534084126 1loss/SRqqt2/assignment_loss‚1.328463251091307 /loss/FBt2/assignment_loss‚0.678987741884258 1loss/SRqqt1/detection_loss‚2.5307811349630356 _runtime‚ 8296.94765255 _step‚910 .loss/FRt1/assignment_loss‚4.64891619682312 /loss/FRt4/detection_loss‚2.1645608596942005 0loss/SRqqt4/detection_loss‚1.717699982225895 % _timestamp‚1.7884156865352483e+09 1loss/SRqqt2/detection_loss‚1.7204396254383028 /loss/FBt1/detection_loss‚2.8775843262672423 /loss/FBt3/detection_loss‚0.7542007037498626 epoch‚9Ž­­†‘ô'‚(:† ÓäÔó–½™ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6726 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºµ÷! õ'‚(:• ÓäÔ庫œ disk.rbd2.in‚0 disk.rbd3.in‚0 disk.sda3.in‚0 disk.sda1.out‚0 proc.memory.rssMB‚ 2888.53125 network.recv‚ 3.063453e+06 disk.sda.out‚ 330.015625 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0# disk.nvme0n1.out‚78068.92578125 proc.cpu.threads‚17 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 715.55859375$ disk./.usageGB‚1097.691967010498 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 4.35546875 disk.dm-1.in‚0 disk.sda3.out‚ 330.015625( proc.memory.availableMB‚ 250.28515625 cpu‚25.390490817445826" disk.nvme0c0n1.in‚ 715.55859375) proc.memory.percent‚17.63019561767578 disk.sda.in‚0$ memory_percent‚98.47238063812256) disk./.usagePercent‚73.66702893098135 disk.dm-0.in‚ 715.55859375 disk.dm-0.out‚78068.92578125! disk.rbd2.out‚202.65869140625% disk.nvme0c0n1.out‚78068.92578125 disk.sda2.out‚0 network.sent‚ 2.6870598e+07 disk.dm-1.out‚ 330.015625ìÉæö'‚(:„ âäԥș™ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.028 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿý/÷'‚(:¤ âäÔà‘› disk.rbd3.in‚0 disk.sda.in‚0 disk.sda.out‚ 330.49609375 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda3.in‚0) proc.memory.percent‚17.63019561767578 disk.nvme0n1.in‚ 715.55859375" disk.nvme0c0n1.in‚ 715.55859375% disk./.usageGB‚1097.5602340698242 disk.dm-0.in‚ 715.55859375 disk.sda3.out‚ 330.49609375 disk.sda1.in‚0( proc.memory.availableMB‚ 250.20703125 network.sent‚ 2.6979423e+07 network.recv‚ 3.071213e+06) disk./.usagePercent‚73.65818822280144 disk.rbd2.in‚0$ memory_percent‚98.47285747528076 disk.dm-1.out‚ 330.49609375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚78168.19384765625 proc.memory.rssMB‚ 2888.53125 proc.cpu.threads‚17 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.35546875& disk.nvme0n1.out‚78168.19384765625! disk.rbd2.out‚204.24853515625 disk.sda2.in‚0 disk.sda2.out‚0# disk.dm-0.out‚78168.19384765625 cpu‚25.38982440189165º7[2ø'- _wandbruntime‚8330  _runtime‚8330R¤X5Èù' & lr-AdamW/pg2‚2.053787131028304e-06 trainer/global_step‚11099 % _timestamp‚1.7884157205016236e+09 _runtime‚8330.913743199 _step‚911 & lr-AdamW/pg1‚2.053787131028304e-06˜C…Íú'Ç trainer/global_step‚11099 % _timestamp‚1.7884157205016236e+09 _runtime‚8330.913743199 _step‚911 & lr-AdamW/pg1‚2.053787131028304e-06 & lr-AdamW/pg2‚2.053787131028304e-06oÍ®2û'- _wandbruntime‚8330  _runtime‚8330Æ7š¼î'ü'èO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚298129381821440 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚313399172120576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚313399154117632 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15664411811328 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313399172120576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚329063752931840 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146597371 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 224604116 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _step‚912 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _runtime‚8330.914795552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146597371 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 148595243 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15937095133696 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚11099 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚332060160003072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63016837 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 224584078 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚329063752931840 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 371201487 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚298129364782080 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15664580811264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 % _timestamp‚1.7884157205024023e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15664411811328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329063565928960 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚347997255136768 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15664580811264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329063565928960 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146597369 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85578406 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15619421067070 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚313748802888510 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146597369 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚347997020612096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 371181447 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚332060136062976 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚524 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15936884549120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 224584078 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985‚žhó'ý'íO UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 224604116 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85578404 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚313748620727746 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329063565928960 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚347997255136768 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15664580811264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329063565928960 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146597369 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85578406 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15619421067070 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚313748802888510 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146597369 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚347997020612096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 371181447 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚332060136062976 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚524 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15936884549120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 224584078 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚298129381821440 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚313399172120576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚313399154117632 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15664411811328 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313399172120576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚329063752931840 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146597371 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 224604116 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _step‚912 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _runtime‚8330.914795552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146597371 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 148595243 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15937095133696 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚11099 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚332060160003072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63016837 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 224584078 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚329063752931840 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 371201487 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚298129364782080 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15664580811264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 % _timestamp‚1.7884157205024023e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15664411811328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 224623929 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚313776135235598 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 371213893 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 224623929 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚329092415448064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚313426423482368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚348027347597312 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 371233959 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 371233959 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146610028 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15938511254528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15620645110210 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚329092228431872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚298155325003776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15665973962752 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15620810231822 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚913 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 371213893 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148607372 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85585781 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15665804949504 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63022114 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146610030 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15938298586112 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚11099 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 $ _timestamp‚1.788415721159054e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85585783 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚313426423482368 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 oü&€(öM FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚348027347597312 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 371233959 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 371233959 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146610028 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15938511254528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15620645110210 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚329092228431872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚298155325003776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15665973962752 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15620810231822 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚913 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 371213893 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148607372 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85585781 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15665804949504 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63022114 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146610030 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15938298586112 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚11099 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 $ _timestamp‚1.788415721159054e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85585783 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚313426423482368 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146610028 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 148607897 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329092415448064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 224603865 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 224623929 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚313776135235598 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 371213893 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 224623929 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚329092415448064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚313426423482368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0‘‚ Þœ2(- _wandbruntime‚8331  _runtime‚8331¥¡•€\‚(Ö _runtime‚8331.572799623 0loss/FRt3/assignment_loss‚2.2218340756941815 2loss/SRqqt3/assignment_loss‚1.3098483734246757 .loss/FRt4/detection_loss‚2.031245988373663 /loss/FRt4/assignment_loss‚3.670161057921017 /loss/FRt1/detection_loss‚3.3811312204315547 /loss/FBt2/detection_loss‚2.3070892776761736 epoch‚9 _step‚914 0loss/FRt2/assignment_loss‚2.8520700396323693 0loss/FBt2/assignment_loss‚1.2866162655742042 1loss/SRqqt3/detection_loss‚1.7693403239051502 /loss/FBt1/detection_loss‚2.5573129745630117 /loss/FBt3/detection_loss‚0.7920653156121261 trainer/global_step‚11099 2loss/SRqqt1/assignment_loss‚2.2740695804357527 0loss/FBt4/assignment_loss‚0.2484361932119403 0loss/SRqqt2/detection_loss‚1.442233221232891 /loss/FRt1/assignment_loss‚4.649033205849784 2loss/SRqqt2/assignment_loss‚1.3243503439938649 2loss/SRqqt4/assignment_loss‚1.4131262461111571 1loss/FBt1/assignment_loss‚0.38953536541885114 1loss/FBt3/assignment_loss‚0.19001683885857687 0loss/SRqqt1/detection_loss‚2.610119829859052 0loss/SRqqt4/detection_loss‚1.771112971007824 /loss/FRt3/detection_loss‚1.7540197635487633 % _timestamp‚1.7884157211607656e+09 /loss/FRt2/detection_loss‚2.1610610363434772 /loss/FBt4/detection_loss‚2.3517052054405214 %loss/total_loss‚46.75753471412059°Fðaƒ(Û /loss/FRt4/assignment_loss‚3.670161057921017 /loss/FRt1/detection_loss‚3.3811312204315547 /loss/FBt2/detection_loss‚2.3070892776761736 epoch‚9 _step‚914 0loss/FRt2/assignment_loss‚2.8520700396323693 0loss/FBt2/assignment_loss‚1.2866162655742042 1loss/SRqqt3/detection_loss‚1.7693403239051502 /loss/FBt1/detection_loss‚2.5573129745630117 /loss/FBt3/detection_loss‚0.7920653156121261 trainer/global_step‚11099 2loss/SRqqt1/assignment_loss‚2.2740695804357527 0loss/FBt4/assignment_loss‚0.2484361932119403 0loss/SRqqt2/detection_loss‚1.442233221232891 /loss/FRt1/assignment_loss‚4.649033205849784 2loss/SRqqt2/assignment_loss‚1.3243503439938649 2loss/SRqqt4/assignment_loss‚1.4131262461111571 1loss/FBt1/assignment_loss‚0.38953536541885114 1loss/FBt3/assignment_loss‚0.19001683885857687 0loss/SRqqt1/detection_loss‚2.610119829859052 0loss/SRqqt4/detection_loss‚1.771112971007824 /loss/FRt3/detection_loss‚1.7540197635487633 % _timestamp‚1.7884157211607656e+09 /loss/FRt2/detection_loss‚2.1610610363434772 /loss/FBt4/detection_loss‚2.3517052054405214 %loss/total_loss‚46.75753471412059 _runtime‚8331.572799623 0loss/FRt3/assignment_loss‚2.2218340756941815 2loss/SRqqt3/assignment_loss‚1.3098483734246757 .loss/FRt4/detection_loss‚2.031245988373663’+îKž‘„(‚(:† ñäÔ•¡›™ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.756' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(hâa-…(‚(:¢ ñäÔ¨Üý› disk.drbd1777.out‚0' disk.nvme0c0n1.out‚78268.4873046875$ memory_percent‚98.47490787506104 disk.sda1.in‚0 disk.nvme0n1.in‚ 715.55859375% disk.nvme0n1.out‚78268.4873046875 disk.dm-0.in‚ 715.55859375 disk.rbd0.in‚0 proc.memory.rssMB‚ 2888.53125 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda.out‚ 331.06640625 disk.rbd1.out‚ 4.35546875" disk.dm-0.out‚78268.4873046875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0) proc.memory.percent‚17.63019561767578 network.sent‚ 2.6985535e+07 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 249.87109375 cpu‚25.389241937392153 disk.drbd1777.in‚0 disk.dm-1.in‚0% disk./.usageGB‚1098.0601654052734 disk.dm-1.out‚ 331.06640625" disk.nvme0c0n1.in‚ 715.55859375 disk.sda3.in‚0 network.recv‚ 3.075524e+06) disk./.usagePercent‚73.69173903419376 disk.sda1.out‚0! disk.rbd2.out‚204.27587890625 disk.sda3.out‚ 331.06640625Ë2SÏ‘†(‚(:† €žäÔÝ®¡™ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.392' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0784 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N®Ú‰#‡(‚(:˜ €žäÔ¬ªâœ disk.rbd1.out‚ 4.35546875 disk.sda3.out‚ 331.46875 disk.sda1.in‚0$ memory_percent‚98.47187995910645 network.recv‚ 3.082407e+06" disk.dm-0.out‚78268.7607421875 disk.rbd3.out‚0 disk.dm-1.out‚ 331.46875 disk.sda2.in‚0 proc.memory.rssMB‚ 2888.53125% disk./.usageGB‚1098.0601654052734 disk.nvme0n1.in‚ 715.55859375% disk.nvme0n1.out‚78268.7607421875 disk.dm-1.in‚0! disk.rbd2.out‚205.99072265625' disk.nvme0c0n1.out‚78268.7607421875 disk.drbd1777.out‚0) disk./.usagePercent‚73.69173903419376 disk.rbd3.in‚0 cpu‚25.388599952179053 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 331.46875 disk.sda1.out‚0' proc.memory.availableMB‚ 250.3671875 proc.cpu.threads‚17 network.sent‚ 2.7093509e+07 disk.rbd1.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 715.55859375 disk.sda.in‚0) proc.memory.percent‚17.63019561767578 disk.dm-0.in‚ 715.55859375 disk.sda3.in‚0 disk.drbd1777.in‚0ß=2ˆ(- _wandbruntime‚8366  _runtime‚8366 |D]ɉ(à ' lr-AdamW/pg2‚1.8530160471867362e-06 trainer/global_step‚11149 $ _timestamp‚1.788415755836273e+09 _runtime‚8366.248407942 _step‚915 ' lr-AdamW/pg1‚1.8530160471867362e-06QñŸBΊ(È ' lr-AdamW/pg1‚1.8530160471867362e-06 ' lr-AdamW/pg2‚1.8530160471867362e-06 trainer/global_step‚11149 $ _timestamp‚1.788415755836273e+09 _runtime‚8366.248407942 _step‚915“«yŒw2‹(- _wandbruntime‚8366  _runtime‚8366½9 b#Œ(èO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚333505807414272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚330496892491264 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 225576780 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚315115433823718 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85947256 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 149233216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _runtime‚8366.249275588 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15734083133952 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚314762640357376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚314762622354432 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15734252133888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚314762622354432 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚11149 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63285406 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15688892885478 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147230321 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 225596818 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚330496705488384 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16008033703424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚349513841117696 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15734252133888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 225596818 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚330496705488384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 372807099 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16007825216000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚314762640357376 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15734252133888 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 225596818 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚330496705488384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 372807099 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16007825216000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚314762640357376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚333505807414272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚330496892491264 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 225576780 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚315115433823718 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85947256 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 149233216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _runtime‚8366.249275588 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15734083133952 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚314762640357376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚314762622354432 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15734252133888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚314762622354432 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚11149 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63285406 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15688892885478 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147230321 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 225596818 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚330496705488384 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16008033703424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚349513841117696 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360”7à'¨2Ž(- _wandbruntime‚8366  _runtime‚8366zö|<÷&(ñM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16009453166592 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚299452467081216 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147242978 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚315142584009834 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚333534720362496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63291565 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 225616631 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚314789891719168 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202182656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚314789909722112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147242980 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 225616631 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226122752 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚299452484120576 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 225596567 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15735645285376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85954631 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 149246198 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 trainer/global_step‚11149 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 372859611 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149245690 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85954633 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚314789909722112 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚333534696422400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚314789891719168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 225596567 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 8366.91891542 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15690282050230 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _step‚917 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147242980 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15690116928618 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚508 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 372859611 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚315142766170806 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚506 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 372839545 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16009250983936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 372839545 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064V«oü&(öM F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 8366.91891542 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15690282050230 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _step‚917 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147242980 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15690116928618 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚508 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 372859611 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚315142766170806 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚506 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 372839545 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16009250983936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 372839545 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16009453166592 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚299452467081216 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147242978 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚315142584009834 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚333534720362496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63291565 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 225616631 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚314789891719168 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202182656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚314789909722112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147242980 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 225616631 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226122752 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚299452484120576 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 225596567 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15735645285376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85954631 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 149246198 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 trainer/global_step‚11149 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 372859611 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149245690 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85954633 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚314789909722112 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚333534696422400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚314789891719168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 225596567 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401•¯‹2‘(- _wandbruntime‚8366  _runtime‚8366XÔ¿Øù’(Ñ /loss/FBt2/detection_loss‚3.6894172032674155 /loss/FRt1/assignment_loss‚4.219318795204162 2loss/SRqqt2/assignment_loss‚1.3015189519957513 1loss/SRqqt2/detection_loss‚1.2980729516696285 0loss/FRt3/assignment_loss‚2.3607994397481282 /loss/FRt4/assignment_loss‚3.494294286990652 1loss/FBt3/assignment_loss‚0.24415059313641552 1loss/SRqqt3/detection_loss‚1.3194429125063694 .loss/FBt4/detection_loss‚2.563675157725811 2loss/SRqqt3/assignment_loss‚1.8771046622513006 trainer/global_step‚11149 _runtime‚8366.920034019 0loss/SRqqt1/detection_loss‚2.607522714754631 .loss/FBt1/detection_loss‚2.600580181926489 epoch‚9 % _timestamp‚1.7884157565079796e+09 /loss/FRt1/detection_loss‚3.2845407366752624 0loss/FBt2/assignment_loss‚0.2411301606340809 0loss/FBt4/assignment_loss‚0.5849848225404154 .loss/FRt3/detection_loss‚2.125304456634654 1loss/SRqqt4/detection_loss‚1.6141875220126793 /loss/FRt4/detection_loss‚1.9399036667784872 /loss/FBt3/detection_loss‚0.7670460953576581 /loss/FRt2/assignment_loss‚2.325969101190567 0loss/SRqqt1/assignment_loss‚2.05473205370122 2loss/SRqqt4/assignment_loss‚1.6815136590453446 _step‚918 0loss/FBt1/assignment_loss‚0.380592612åÅØf^4210828 .loss/FRt2/detection_loss‚2.034098679423332 %loss/total_loss‚46.60990141759154 T_Ö\“(Ö 0loss/FBt1/assignment_loss‚0.3805926124210828 .loss/FRt2/detection_loss‚2.034098679423332 %loss/total_loss‚46.60990141759154 /loss/FBt2/detection_loss‚3.6894172032674155 /loss/FRt1/assignment_loss‚4.219318795204162 2loss/SRqqt2/assignment_loss‚1.3015189519957513 1loss/SRqqt2/detection_loss‚1.2980729516696285 0loss/FRt3/assignment_loss‚2.3607994397481282 /loss/FRt4/assignment_loss‚3.494294286990652 1loss/FBt3/assignment_loss‚0.24415059313641552 1loss/SRqqt3/detection_loss‚1.3194429125063694 .loss/FBt4/detection_loss‚2.563675157725811 2loss/SRqqt3/assignment_loss‚1.8771046622513006 trainer/global_step‚11149 _runtime‚8366.920034019 0loss/SRqqt1/detection_loss‚2.607522714754631 .loss/FBt1/detection_loss‚2.600580181926489 epoch‚9 % _timestamp‚1.7884157565079796e+09 /loss/FRt1/detection_loss‚3.2845407366752624 0loss/FBt2/assignment_loss‚0.2411301606340809 0loss/FBt4/assignment_loss‚0.5849848225404154 .loss/FRt3/detection_loss‚2.125304456634654 1loss/SRqqt4/detection_loss‚1.6141875220126793 /loss/FRt4/detection_loss‚1.9399036667784872 /loss/FBt3/detection_loss‚0.7670460953576581 /loss/FRt2/assignment_loss‚2.325969101190567 0loss/SRqqt1/assignment_loss‚2.05473205370122 2loss/SRqqt4/assignment_loss‚1.6815136590453446 _step‚918–éé‹‘”(‚(:† žäÔ‚Ê™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚120.671' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1342 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼ Mô•(‚(: žäÔûÈœ disk.rbd0.in‚0 disk.sda.out‚332.1875) disk./.usagePercent‚73.69173903419376& disk.nvme0c0n1.out‚78368.513671875 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 715.55859375$ memory_percent‚98.47383499145508 disk.nvme0n1.in‚ 715.55859375 disk.rbd3.in‚0 network.sent‚ 2.7099003e+07 disk.sda2.out‚0 disk.sda3.out‚332.1875 proc.cpu.threads‚17 disk.sda3.in‚0 cpu‚25.38802600814634 network.recv‚ 3.085972e+06 disk.dm-1.out‚332.1875 disk.rbd0.out‚0 disk.rbd2.in‚0! disk.dm-0.out‚78368.513671875% disk./.usageGB‚1098.0601654052734 disk.drbd1777.in‚0$ disk.nvme0n1.out‚78368.513671875) proc.memory.percent‚17.63019561767578 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 4.35546875 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 715.55859375 disk.dm-1.in‚0! disk.rbd2.out‚206.01416015625 disk.sda2.in‚0& proc.memory.availableMB‚ 250.046875 proc.memory.rssMB‚ 2888.53125/E'‘–(‚(:† žžäÔ±ü²™ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.163' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï¸ÖA&—(‚(:› žžäԮĜœ disk.sda.out‚ 332.55078125 disk.sda1.in‚0! disk.dm-0.out‚78467.677734375$ disk./.usageGB‚1098.060287475586 disk.sda1.out‚0 disk.rbd1.out‚ 4.35546875$ disk.nvme0n1.out‚78467.677734375 proc.cpu.threads‚17) disk./.usagePercent‚73.69174722643486 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0& disk.nvme0c0n1.out‚78467.677734375& proc.memory.availableMB‚ 249.796875) proc.memory.percent‚17.63019561767578 network.sent‚ 2.7208056e+07 disk.sda3.out‚ 332.55078125 disk.sda.in‚0 network.recv‚ 3.093836e+06 disk.rbd1.in‚0 disk.nvme0n1.in‚ 715.55859375 disk.dm-1.in‚0 disk.dm-1.out‚ 332.55078125 disk.drbd1777.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 715.55859375 disk.rbd0.out‚0! disk.rbd2.out‚207.66650390625 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 715.55859375 proc.memory.rssMB‚ 2888.53125 cpu‚25.38739544158615$ memory_percent‚98.47536087036133 disk.rbd0.in‚0 disk.rbd2.in‚0Fs¿Ðœ˜(‚(:‘ ­žäÔÕÊ¢™ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.799' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.159800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`‚®€+™(‚(:  ­žäÔŒ‰•œ$ disk./.usageGB‚1098.060287475586% disk.nvme0n1.out‚78566.8291015625 disk.rbd2.in‚0 disk.sda2.in‚0 cpu‚25.386795067821904 disk.nvme0n1.in‚ 715.55859375 disk.dm-0.in‚ 715.55859375 disk.dm-1.in‚0 disk.dm-1.out‚ 333.33984375" disk.nvme0c0n1.in‚ 715.55859375 disk.drbd1777.in‚0) proc.memory.percent‚17.63019561767578" disk.dm-0.out‚78566.8291015625' proc.memory.availableMB‚ 249.7890625 network.sent‚ 2.7214222e+07 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda.out‚ 333.33984375 disk.sda1.in‚0 disk.rbd1.out‚ 4.35546875! disk.rbd2.out‚207.69384765625 disk.rbd3.out‚0' disk.nvme0c0n1.out‚78566.8291015625 disk.drbd1777.out‚0 disk.sda3.out‚ 333.33984375 disk.sda1.out‚0 network.recv‚ 3.098475e+06 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda3.in‚0$ memory_percent‚98.47540855407715 proc.cpu.threads‚17) disk./.usagePercent‚73.69174722643486 proc.memory.rssMB‚ 2888.53125p3ºß2š(- _wandbruntime‚8401  _runtime‚8401V~>Ê›(Ä ' lr-AdamW/pg1‚1.6623865454278503e-06 ' lr-AdamW/pg2‚1.6623865454278503e-06 trainer/global_step‚11199 % _timestamp‚1.7884157911535213e+09 _runtime‚8401.565619378 _step‚919ÂrÑÏœ(É ' lr-AdamW/pg1‚1.6623865454278503e-06 ' lr-AdamW/pg2‚1.6623865454278503e-06 trainer/global_step‚11199 % _timestamp‚1.7884157911535213e+09 _runtime‚8401.565619378 _step‚919—‹òÐÅ2(- _wandbruntime‚8401  _runtime‚8401o9Êûí'ž(çO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 149872553 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚334951454825472 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚331929845047808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚316482064758926 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚316126108594176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚351030082249216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚316126108594176 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15803923456512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 trainer/global_step‚11199 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15758199582482 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147863271 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63557001 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 149873107 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚300723700055040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86316106 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86316104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15803754456576 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚331929845047808 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚316481882598162 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15758364703886 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 226569482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _runtime‚8401.566551795 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚300723683015680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚316126090591232 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147863269 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16078651363840 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416Øïžzò'Ÿ(ìO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16078864045568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147863271 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚334951430885376 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚331930032050688 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚316481882598162 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15758364703886 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 226569482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _runtime‚8401.566551795 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚300723683015680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚316126090591232 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147863269 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16078651363840 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 149872553 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚334951454825472 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚331929845047808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚316482064758926 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚316126108594176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚351030082249216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚316126108594176 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15803923456512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 trainer/global_step‚11199 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15758199582482 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147863271 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63557001 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 149873107 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚300723700055040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86316106 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86316104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15803754456576 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚331929845047808 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374485263 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 226589269 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147875928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86323483 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 226609333 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15805147594752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147875928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚316509397106014 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚331958507550720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 trainer/global_step‚11199 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚550 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63562280 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 374465197 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 374465197 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15759753868638 _runtime‚8402.228714896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚331958694566912 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚300749626198016 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 % _timestamp‚1.7884157918160033e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚316153377958912 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 149885763 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149885213 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 374485263 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16080073084928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚300749626198016 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 % _timestamp‚1.7884157918160033e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚316153377958912 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 149885763 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149885213 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 374485263 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16080073084928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374485263 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 226589269 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147875928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86323483 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 226609333 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15805147594752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147875928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚316509397106014 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚331958507550720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 trainer/global_step‚11199 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚550 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63562280 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 374465197 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 374465197 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15759753868638 _runtime‚8402.228714896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚331958694566912 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333362984607232 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15873425779200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚317489576830976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63826823 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚317489558828032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 148496219 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚333363171610112 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 trainer/global_step‚11249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚333363171610112 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 150511263 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚317848513533370 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 227582222 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚302020842132480 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15827836522294 $ _timestamp‚1.788415826597575e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15873425779200 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86684956 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚514 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚352546805368320 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86684954 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150511779 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16149494644224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 227562184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚336397102236672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚352546572940800 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 376058403 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148496221 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚317489576830976 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15873594779136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚516 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16149703131648 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _step‚924 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148496219 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚317489558828032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 227562184 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336397078296576 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚333362984607232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 376078443 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15827671400890 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15873594779136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63826309 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚302020859171840 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 227582222 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 trainer/global_step‚11249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚333363171610112 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 150511263 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚317848513533370 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 227582222 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚302020842132480 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15827836522294 $ _timestamp‚1.788415826597575e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15873425779200 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86684956 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚514 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚352546805368320 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86684954 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150511779 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16149494644224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 227562184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚336397102236672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚352546572940800 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 376058403 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148496221 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚317489576830976 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15873594779136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚516 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16149703131648 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _step‚924 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148496219 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚317489558828032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 227562184 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336397078296576 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚333362984607232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 376078443 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15827671400890 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15873594779136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63826309 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚302020859171840 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 227582222 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333362984607232 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15873425779200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚317489576830976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63826823 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚317489558828032œrQál2°(- _wandbruntime‚8437  _runtime‚8437G¨.™ø&±(òM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚302046785314816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15829060565434 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150523921 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚333391647110144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376110915 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148508880 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15874987930624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63832088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚317875845880250 trainer/global_step‚11249 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15874818917376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148508880 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 376110915 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86692333 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 % _timestamp‚1.7884158272777894e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚317516828192768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚352576902842368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚317516846195712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚317516846195712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 376090849 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 227602035 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 227581971 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚317516828192768y£˜ý&²(÷M MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚317516828192768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚352576902842368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚317516846195712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚317516846195712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 376090849 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 227602035 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 227581971 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚317516828192768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚302046785314816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15829060565434 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150523921 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚333391647110144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376110915 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148508880 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15874987930624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63832088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚317875845880250 trainer/global_step‚11249 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15874818917376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148508880 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 376110915 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86692333 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 % _timestamp‚1.7884158272777894e+09&x&2³(- _wandbruntime‚8437  _runtime‚8437è ¨ÀV´(Ð /loss/FRt3/assignment_loss‚2.119252678649179 1loss/SRqqt1/assignment_loss‚1.988771276978346 1loss/FBt3/assignment_loss‚0.13816252261901932 .loss/FRt1/detection_loss‚2.962085051669015 epoch‚9 _step‚926 0loss/FBt1/assignment_loss‚0.2807723503300132 /loss/SRqqt1/detection_loss‚2.72991082072258 0loss/FRt2/assignment_loss‚3.5282532739639283 0loss/FBt2/assignment_loss‚0.4630799314743074 /loss/FRt2/detection_loss‚2.2714163675904273 .loss/FRt1/assignment_loss‚4.66312270032035 2loss/SRqqt2/assignment_loss‚1.2682675943679866 2loss/SRqqt3/assignment_loss‚1.7598675638437271 .loss/FRt4/detection_loss‚2.012702204433142 0loss/SRqqt2/detection_loss‚1.712295905236275 .loss/FBt4/detection_loss‚2.449513538317247 %loss/total_loss‚48.90537855289587 /loss/FRt3/detection_loss‚1.9092647079378366 /loss/FBt2/detection_loss‚3.3396726080349515 /loss/FBt3/detection_loss‚0.8709840612246896 2loss/SRqqt4/assignment_loss‚1.6767493894696235 1loss/SRqqt4/detection_loss‚1.9837265384197236 _runtime‚8437.691434406 /loss/FRt4/assignment_loss‚3.667756596032311 trainer/global_step‚11249 1loss/FBt4/assignment_loss‚0.47717692449590354 1loss/SRqqt3/detection_loss‚1.7762515390428102 .loss/FBt1/detection_loss‚2.856322407722473 % _timestamp‚1.7884158272794037e+09tô!ì[µ(Õ /loss/FRt4/assignment_loss‚3.667756596032311 trainer/global_step‚11249 1loss/FBt4/assignment_loss‚0.47717692449590354 1loss/SRqqt3/detection_loss‚1.7762515390428102 .loss/FBt1/detection_loss‚2.856322407722473 % _timestamp‚1.7884158272794037e+09 /loss/FRt3/assignment_loss‚2.119252678649179 1loss/SRqqt1/assignment_loss‚1.988771276978346 1loss/FBt3/assignment_loss‚0.13816252261901932 .loss/FRt1/detection_loss‚2.962085051669015 epoch‚9 _step‚926 0loss/FBt1/assignment_loss‚0.2807723503300132 /loss/SRqqt1/detection_loss‚2.72991082072258 0loss/FRt2/assignment_loss‚3.5282532739639283 0loss/FBt2/assignment_loss‚0.4630799314743074 /loss/FRt2/detection_loss‚2.2714163675904273 .loss/FRt1/assignment_loss‚4.66312270032035 2loss/SRqqt2/assignment_loss‚1.2682675943679866 2loss/SRqqt3/assignment_loss‚1.7598675638437271 .loss/FRt4/detection_loss‚2.012702204433142 0loss/SRqqt2/detection_loss‚1.712295905236275 .loss/FBt4/detection_loss‚2.449513538317247 %loss/total_loss‚48.90537855289587 /loss/FRt3/detection_loss‚1.9092647079378366 /loss/FBt2/detection_loss‚3.3396726080349515 /loss/FBt3/detection_loss‚0.8709840612246896 2loss/SRqqt4/assignment_loss‚1.6767493894696235 1loss/SRqqt4/detection_loss‚1.9837265384197236 _runtime‚8437.691434406ž›Fs‘¶(‚(:† ÚžäÔ–»¹™ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.557' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1114 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y‘À)·(‚(:ž ÚžäÔ disk.sda.out‚ 335.10546875" disk.dm-0.out‚78766.2685546875" disk.nvme0c0n1.in‚ 715.55859375 disk.sda2.out‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚78766.2685546875 disk.dm-0.in‚ 715.55859375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 cpu‚25.384974983442635 disk.sda3.out‚ 335.10546875# memory_percent‚98.4684944152832 network.sent‚ 2.7334015e+07 disk.rbd1.out‚ 4.35546875! disk.rbd2.out‚210.95947265625 disk.rbd3.in‚0) proc.memory.percent‚17.63019561767578 network.recv‚ 3.113714e+06) disk./.usagePercent‚73.69174722643486 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 715.55859375 disk.dm-1.out‚ 335.10546875 disk.sda2.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.rbd3.out‚0' disk.nvme0c0n1.out‚78766.2685546875 disk.sda3.in‚0& proc.memory.availableMB‚ 250.921875 proc.memory.rssMB‚ 2888.53125$ disk./.usageGB‚1098.060287475586ëïMÿ¸(‚(:† éžäÔ¨¨£™ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.339' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.úÉM|’powerPercent‚23.2678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚îøø,¹(‚(:¡ éžäÔড় network.recv‚ 3.121268e+06% disk./.usageGB‚1098.0603103637695 disk.sda1.out‚0 disk.rbd2.in‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 715.55859375 disk.sda2.out‚0# memory_percent‚98.4684705734253 disk.drbd1777.in‚0% disk.nvme0n1.out‚78766.8876953125 disk.dm-0.in‚ 715.55859375) proc.memory.percent‚17.63019561767578 disk.sda.in‚0 disk.sda1.in‚0" disk.dm-0.out‚78766.8876953125 disk.dm-1.in‚0 disk.dm-1.out‚ 335.75390625' disk.nvme0c0n1.out‚78766.8876953125 disk.sda3.out‚ 335.75390625 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.rbd2.out‚212.49853515625 proc.cpu.threads‚17 network.sent‚ 2.7442766e+07 disk.sda.out‚ 335.75390625 disk.rbd1.in‚0 disk.sda2.in‚0 proc.memory.rssMB‚ 2888.53125) disk./.usagePercent‚73.69174876248006 disk.drbd1777.out‚0 disk.rbd1.out‚ 4.35546875 disk.nvme0n1.in‚ 715.55859375 disk.rbd3.out‚0( proc.memory.availableMB‚ 250.92578125 cpu‚25.384353594312543 disk.sda3.in‚0T2º(- _wandbruntime‚8471  _runtime‚8471¾ú)¯Ê»(Ä ' lr-AdamW/pg2‚1.3117090739983474e-06 trainer/global_step‚11299 % _timestamp‚1.7884158613898582e+09 _runtime‚8471.801986439 _step‚927 ' lr-AdamW/pg1‚1.3117090739983474e-06~RGϼ(É ' lr-AdamW/pg1‚1.3117090739983474e-06 ' lr-AdamW/pg2‚1.3117090739983474e-06 trainer/global_step‚11299 % _timestamp‚1.7884158613898582e+09 _runtime‚8471.801986439 _step‚927Ÿ¯\x 2½(- _wandbruntime‚8471  _runtime‚8471Wk°hî'¾(èO O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 228574924 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15943266101760 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64094839 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚354063070597632 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 _step‚928 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 228574924 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15897308340702 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15897143219298 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚334796124166656 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15943097101824 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15943097101824 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚11299 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149129171 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚318853027064832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16220555474432 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚319215326629342 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚318853045067776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 228554886 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 _runtime‚8471.802936249 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 377684055 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚334796311169536 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15943266101760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 % _timestamp‚1.7884158613905945e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149129171 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87053804 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16220344889856 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚303318018288640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚334796311169536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚318853027064832 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚334796124166656 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 377704095 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149129169 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚354063305122304 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚303318001249280 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚499é}¥)ó'¿(íO UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149129171 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87053804 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16220344889856 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚303318018288640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚334796311169536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚318853027064832 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚334796124166656 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 377704095 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149129169 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚354063305122304 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚303318001249280 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚499 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 228574924 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15943266101760 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64094839 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚354063070597632 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 _step‚928 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 228574924 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15897308340702 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15897143219298 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚334796124166656 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15943097101824 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15943097101824 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚11299 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149129171 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚318853027064832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16220555474432 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚319215326629342 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚318853045067776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 228554886 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 _runtime‚8471.802936249 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 377684055 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚334796311169536 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15943266101760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 % _timestamp‚1.7884158613905945e+09 ¬­I<2À(- _wandbruntime‚8472  _runtime‚8472×å¸ø&Á(òM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚334824786669568 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149141830 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16221973659648 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151160869 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64100193 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚505 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚337871638656000 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚337871662596096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16221754699776 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 377736567 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚318880314432512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 377736567 _runtime‚8472.681008407 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149141830 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚334824973685760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 377716501 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚929 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64099688 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚318880296429568 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15944490240000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149141828 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15898532383842 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149141828 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 228594737 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 228574673 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚318880314432512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚354093393355776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚319242476815458 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 228574673 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15944490240000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 trainer/global_step‚11299 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15944659253248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚334824973685760 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 228594737 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87061183 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15944659253248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚334824973685760 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 228594737 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87061183 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151160869 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64100193 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚505 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚337871638656000 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚337871662596096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16221754699776 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 377736567 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚318880314432512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 377736567 _runtime‚8472.681008407 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149141830 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚334824973685760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 377716501 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚929 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64099688 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚318880296429568 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15944490240000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149141828 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15898532383842 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149141828 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 228594737 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 228574673 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚318880314432512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚354093393355776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚319242476815458 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¡ØÚÐv2Ã(- _wandbruntime‚8472  _runtime‚8472¾rƒY\Ä(Ö 0loss/FBt2/assignment_loss‚0.3168588122964138 1loss/FBt4/assignment_loss‚0.45998165586675704 .loss/FRt1/detection_loss‚3.308940244757611 /loss/FRt4/assignment_loss‚3.821497270796034 1loss/SRqqt1/assignment_loss‚2.467036642738291 1loss/SRqqt1/detection_loss‚2.4573316042487687 /loss/FRt2/detection_loss‚2.1952394907889157 2loss/SRqqt3/assignment_loss‚1.2526243063979423 2loss/SRqqt4/assignment_loss‚1.4801826116650603 1loss/FBt1/assignment_loss‚0.44782947794986905 1loss/SRqqt2/detection_loss‚1.4434654414653778 epoch‚9 2loss/SRqqt2/assignment_loss‚1.4017672075918226 /loss/FBt2/detection_loss‚3.1248245537281036 .loss/FBt1/detection_loss‚2.387082187752975 0loss/FRt2/assignment_loss‚2.8609622703946154 /loss/FRt3/detection_loss‚1.9258759274732236 _step‚930 .loss/FRt4/detection_loss‚2.299202327595817 1loss/SRqqt3/detection_loss‚1.2636184059083462 1loss/SRqqt4/detection_loss‚2.0964903280485507 /loss/FBt3/detection_loss‚0.7483012489141019 0loss/FBt3/assignment_loss‚0.2648139524524437 /loss/FRt1/assignment_loss‚4.202364755713421 0loss/FRt3/assignment_loss‚1.7583428773769112 /loss/FBt4/detection_loss‚2.3990062334035573 %loss/total_loss‚46.38363983532493 trainer/global_step‚11299 _runtime‚8472.682147994 % _timestamp‚1.7884158622701244e+09,ÒˆaÅ(Û .loss/FBt1/detection_loss‚2.387082187752975 0loss/FRt2/assignment_loss‚2.8609622703946154 /loss/FRt3/detection_loss‚1.9258759274732236 _step‚930 .loss/FRt4/detection_loss‚2.299202327595817 1loss/SRqqt3/detection_loss‚1.2636184059083462 1loss/SRqqt4/detection_loss‚2.0964903280485507 /loss/FBt3/detection_loss‚0.7483012489141019 0loss/FBt3/assignment_loss‚0.2648139524524437 /loss/FRt1/assignment_loss‚4.202364755713421 0loss/FRt3/assignment_loss‚1.7583428773769112 /loss/FBt4/detection_loss‚2.3990062334035573 %loss/total_loss‚46.38363983532493 trainer/global_step‚11299 _runtime‚8472.682147994 % _timestamp‚1.7884158622701244e+09 0loss/FBt2/assignment_loss‚0.3168588122964138 1loss/FBt4/assignment_loss‚0.45998165586675704 .loss/FRt1/detection_loss‚3.308940244757611 /loss/FRt4/assignment_loss‚3.821497270796034 1loss/SRqqt1/assignment_loss‚2.467036642738291 1loss/SRqqt1/detection_loss‚2.4573316042487687 /loss/FRt2/detection_loss‚2.1952394907889157 2loss/SRqqt3/assignment_loss‚1.2526243063979423 2loss/SRqqt4/assignment_loss‚1.4801826116650603 1loss/FBt1/assignment_loss‚0.44782947794986905 1loss/SRqqt2/detection_loss‚1.4434654414653778 epoch‚9 2loss/SRqqt2/assignment_loss‚1.4017672075918226 /loss/FBt2/detection_loss‚3.1248245537281036¢jHWuœÆ(‚(:‘ øžäÔ¸þË™ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.799' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.159800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄÁ,%Ç(‚(:š øžäԙτž disk.rbd1.in‚0 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 715.55859375) proc.memory.percent‚17.63019561767578 disk.sda.out‚ 336.203125 disk.nvme0n1.in‚ 715.55859375 disk.dm-0.in‚ 715.55859375 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 proc.memory.rssMB‚ 2888.53125 disk.sda.in‚0 disk.drbd1777.out‚0$ memory_percent‚98.47157001495361% disk./.usageGB‚1098.0602912902832" disk.dm-0.out‚78865.9658203125' disk.nvme0c0n1.out‚78865.9658203125 disk.sda2.out‚0 disk.sda3.out‚ 336.203125( proc.memory.availableMB‚ 250.41796875 network.recv‚ 3.125066e+06( disk./.usagePercent‚73.6917474824424 disk.dm-1.out‚ 336.203125 cpu‚25.38375275786855 network.sent‚ 2.7448085e+07 disk.drbd1777.in‚0 disk.rbd1.out‚ 4.44140625% disk.nvme0n1.out‚78865.9658203125! disk.rbd2.out‚212.63916015625 disk.sda2.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd3.in‚0³š„ë‘È(‚(:† ‡ŸäÔ˸À™ gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.194' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8388 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03)â(É(‚(: ‡ŸäԭɃœ disk.rbd2.in‚0! disk.rbd2.out‚214.17431640625 disk.sda.out‚ 336.9921875$ memory_percent‚98.46868515014648 cpu‚25.38314786780142& disk.nvme0n1.out‚78965.18798828125 network.recv‚ 3.131704e+06 disk.rbd3.in‚0( disk.nvme0c0n1.out‚78965.18798828125 disk.sda3.out‚ 336.9921875 disk.sda1.in‚0 disk.rbd1.out‚ 4.44140625$ disk./.usageGB‚1098.059970855713 disk.dm-1.out‚ 336.9921875 disk.sda3.in‚0 disk.rbd1.in‚0# disk.dm-0.out‚78965.18798828125 disk.dm-1.in‚0 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 715.55859375& proc.memory.availableMB‚ 250.890625 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 715.55859375 disk.sda2.in‚0) proc.memory.percent‚17.63019561767578 network.sent‚ 2.755651e+07) disk./.usagePercent‚73.69172597780951 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.dm-0.in‚ 715.55859375 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.memory.rssMB‚ 2888.53125 proc.cpu.threads‚17«ÂÎ7œÊ(‚(:‘ –ŸäÔÛ€™ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.526' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.705199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íK [/Ë(‚(:¤ –ŸäÔÔãÇ›) disk./.usagePercent‚73.69172597780951 disk.rbd2.in‚0 proc.cpu.threads‚17 network.sent‚ 2.7562763e+07 disk.sda.in‚0# disk.dm-0.out‚78965.44970703125! disk.rbd2.out‚214.31494140625 network.recv‚ 3.136735e+06 disk.sda1.out‚0 disk.drbd1777.out‚0" disk.nvme0c0n1.in‚ 715.55859375 disk.sda2.out‚0 disk.sda.out‚ 337.35546875 disk.dm-0.in‚ 715.55859375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚78965.44970703125 proc.memory.rssMB‚ 2888.53125 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 715.55859375& disk.nvme0n1.out‚78965.44970703125 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0) proc.memory.percent‚17.63019561767578 cpu‚25.382564927067303$ disk./.usageGB‚1098.059970855713 disk.sda3.out‚ 337.35546875 disk.rbd1.out‚ 4.50390625 disk.dm-1.out‚ 337.35546875$ memory_percent‚98.47023487091064( proc.memory.availableMB‚ 250.63671875 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda2.in‚0ÅLñ¦2Ì(- _wandbruntime‚8507  _runtime‚8507n_øÊÍ(Ä _step‚931 ' lr-AdamW/pg1‚1.1517341849871786e-06 ' lr-AdamW/pg2‚1.1517341849871786e-06 trainer/global_step‚11349 % _timestamp‚1.7884158966900017e+09 _runtime‚8507.102237014ûB«êÏÎ(É _step‚931 ' lr-AdamW/pg1‚1.1517341849871786e-06 ' lr-AdamW/pg2‚1.1517341849871786e-06 trainer/global_step‚11349 % _timestamp‚1.7884158966900017e+09 _runtime‚8507.102237014£±;hì2Ï(- _wandbruntime‚8507  _runtime‚8507^Üî'Ð(èO N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151787575 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16291180210688 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15966615037706 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149762119 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149762119 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚320216495301632 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚336229263726080 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87422654 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚304615160366080 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884158966907456e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64365460 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚336229450728960 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16012768424448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16012937424384 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15966780159110 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚339288373118976 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚339288397059072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149762121 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 229547588 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚336229450728960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87422656 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚320216495301632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 229547588 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151788116 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚320581957564550 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64364921 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚320581775403786 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚304615177405440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16012768424448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚355579787854336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚355579553329664 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚11349 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚320216513304576 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚336229263726080 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 379329747 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16012937424384 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16291390795264 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _step‚932 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149762121ÉÒ'®aÑ(íO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149762119 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚320216495301632 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚336229263726080 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87422654 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚304615160366080 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884158966907456e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64365460 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚336229450728960 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16012768424448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16012937424384 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15966780159110 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚339288373118976 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚339288397059072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149762121 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 229547588 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚336229450728960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87422656 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚320216495301632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 229547588 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151788116 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚320581957564550 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64364921 =7DevicH5`õ’eStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚320581775403786 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚304615177405440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16012768424448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚355579787854336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚355579553329664 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚11349 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚320216513304576 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚336229263726080 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 379329747 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16012937424384 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16291390795264 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _step‚932 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149762121 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151787575 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16291180210688 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15966615037706 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149762119 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096¤N› 2Ò(- _wandbruntime‚8507  _runtime‚8507Vù)÷&Ó(ñM OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 229567375 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15968004202250 _runtime‚8507.762001431 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 379362219 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚320243764666368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚933 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚320243764666368 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16014330575872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149774780 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16014161562624 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15968169323862 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 229567375 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16014161562624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 379342153 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 379342153 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 229587439 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚355609882255360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚320609107750666 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 $ _timestamp‚1.788415897349142e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149774780 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149774778 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚320243782669312 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚339317286067200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚320243782669312 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16292806759424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚304641120587776 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚320609289911638 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚336258113245184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚11349 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚336257926228992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151800270 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚498 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64370239 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 379362219 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 229587439 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64370735 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚336258113245184{±…ü&Ô(öM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 379362219 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚320243764666368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚933 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚320243764666368 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16014330575872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149774780 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16014161562624 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15968169323862 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 229567375 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16014161562624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 379342153 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 379342153 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 229587439 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚355609882255360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚320609107750666 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 $ _timestamp‚1.788415897349142e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149774780 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149774778 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚320243782669312 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚339317286067200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚320243782669312 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16292806759424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚304641120587776 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚320609289911638 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚336258113245184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚11349 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚336257926228992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151800270 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚498 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64370239 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 379362219 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 229587439 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64370735 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚336258113245184 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 229567375 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15968004202250 _runtime‚8507.762001431 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217¥µÒñ2Õ(- _wandbruntime‚8507  _runtime‚8507®þ‹eTÖ(Î epoch‚9 0loss/FRt3/assignment_loss‚1.8184863603115082 0loss/SRqqt3/detection_loss‚1.513801808132136 /loss/FBt2/detection_loss‚2.3800589186804637 /loss/FBt3/detection_loss‚0.7308374375891076 1loss/SRqqt3/assignment_loss‚1.117691238311694 1loss/SRqqt4/detection_loss‚1.8274735760688783 /loss/FBt4/detection_loss‚2.3131792843341827 % _timestamp‚1.7884158973515604e+09 /loss/FRt2/assignment_loss‚2.618893069464986 trainer/global_step‚11349 .loss/FRt2/detection_loss‚2.631988658410747 1loss/SRqqt1/assignment_loss‚2.434811905736015 2loss/SRqqt4/assignment_loss‚1.3460604210197926 0loss/FBt3/assignment_loss‚0.3166711692119271 0loss/FBt4/assignment_loss‚0.6954316417047721 .loss/FRt3/detection_loss‚1.948089514374733 -loss/FBt1/detection_loss‚3.00032517043027 0loss/FRt4/assignment_loss‚3.6928180319567523 0loss/FBt1/assignment_loss‚0.6118933375037159 /loss/FRt4/detection_loss‚2.1633605125049757 0loss/SRqqt1/detection_loss‚2.968454358123598 $loss/total_loss‚47.6608627823231 0loss/FBt2/assignment_loss‚0.6060346135574426 .loss/FRt1/detection_loss‚3.023159466443523 1loss/SRqqt2/detection_loss‚1.8009036335128326 /loss/FRt1/assignment_loss‚4.806096084656254 2loss/SRqqt2/assignment_loss‚1.2943425702827949 _runtime‚8507.763664537 _step‚934ÄèŃY×(Ó 1loss/SRqqt1/assignment_loss‚2.434811905736015 2loss/SRqqt4/assignment_loss‚1.3460604210197926 0loss/FBt3/assignment_loss‚0.3166711692119271 0loss/FBt4/assignment_loss‚0.6954316417047721 .loss/FRt3/detection_loss‚1.948089514374733 -loss/FBt1/detection_loss‚3.00032517043027 0loss/FRt4/assignment_loss‚3.6928180319567523 0loss/FBt1/assignment_loss‚0.6118933375037159 /loss/FRt4/detection_loss‚2.1633605125049757 0loss/SRqqt1/detection_loss‚2.968454358123598 $loss/total_loss‚47.6608627823231 0loss/FBt2/assignment_loss‚0.6060346135574426 .loss/FRt1/detection_loss‚3.023159466443523 1loss/SRqqt2/detection_loss‚1.8009036335128326 /loss/FRt1/assignment_loss‚4.806096084656254 2loss/SRqqt2/assignment_loss‚1.2943425702827949 _runtime‚8507.763664537 _step‚934 epoch‚9 0loss/FRt3/assignment_loss‚1.8184863603115082 0loss/SRqqt3/detection_loss‚1.513801808132136 /loss/FBt2/detection_loss‚2.3800589186804637 /loss/FBt3/detection_loss‚0.7308374375891076 1loss/SRqqt3/assignment_loss‚1.117691238311694 1loss/SRqqt4/detection_loss‚1.8274735760688783 /loss/FBt4/detection_loss‚2.3131792843341827 % _timestamp‚1.7884158973515604e+09 /loss/FRt2/assignment_loss‚2.618893069464986 trainer/global_step‚11349 .loss/FRt2/detection_loss‚2.631988658410747¦#ëœØ(‚(:‘ ¥ŸäÔ×üÉ™ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.066' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.813200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y3eª(Ù(‚(: ¥ŸäÔ™´´œ% disk.nvme0n1.out‚79064.7666015625" disk.dm-0.out‚79064.7666015625# memory_percent‚98.4717845916748 disk.drbd1777.out‚0) proc.memory.percent‚17.63019561767578 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd1777.in‚0 network.sent‚ 2.7568904e+07% disk./.usageGB‚1098.0610733032227 disk.sda2.in‚0 disk.dm-0.in‚ 715.55859375 disk.sda2.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 4.50390625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚0' proc.memory.availableMB‚ 250.3828125 network.recv‚ 3.141476e+06) disk./.usagePercent‚73.69179996398691 disk.sda3.in‚0 disk.sda.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 715.55859375 disk.rbd2.in‚0' disk.nvme0c0n1.out‚79064.7666015625 cpu‚25.381946554922315 disk.sda3.out‚ 337.8984375 proc.memory.rssMB‚ 2888.53125 proc.cpu.threads‚17 disk.sda.out‚ 337.8984375 disk.dm-1.out‚ 337.8984375! disk.rbd2.out‚215.85400390625" disk.nvme0c0n1.in‚ 715.55859375 Þ¤ZÚ(‚(:… ´ŸäÔâÞÙ gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚98.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.6716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ˆÈ²Û(‚(:ú ´ŸäÔðéœ disk.dm-0.in‚ 715.55859375 disk.rbd0.out‚0! disk.rbd2.out‚217.56884765625 disk.sda2.out‚0% disk./.usageGB‚1098.0610961914062 disk.dm-1.out‚338.5 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 715.55859375 network.sent‚ 2.7677085e+07 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda.out‚338.5 disk.nvme0n1.out‚ 79163.875( proc.memory.availableMB‚ 250.35546875 disk.dm-0.out‚ 79163.875 disk.rbd3.in‚0$ memory_percent‚98.47195148468018 cpu‚25.381365394245957 network.recv‚ 3.147887e+06 disk.rbd0.in‚0 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 715.55859375 disk.nvme0c0n1.out‚ 79163.875 disk.sda2.in‚0 disk.sda3.out‚338.5 disk.rbd1.out‚ 4.5361328125) disk./.usagePercent‚73.69180150003211 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.out‚0 proc.memory.rssMB‚ 2888.53125 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda3.in‚0) proc.memory.percent‚17.63019561767578—u;2Ü(- _wandbruntime‚8542  _runtime‚8542'ERÊÝ(Ä _step‚935 ' lr-AdamW/pg1‚1.0020470241476586e-06 ' lr-AdamW/pg2‚1.0020470241476586e-06 trainer/global_step‚11399 % _timestamp‚1.7884159318688092e+09 _runtime‚8542.280963912;ûÏÞ(É ' lr-AdamW/pg2‚1.0020470241476586e-06 trainer/global_step‚11399 % _timestamp‚1.7884159318688092e+09 _runtime‚8542.280963912 _step‚935 ' lr-AdamW/pg1‚1.0020470241476586e-06§ã¢Y2ß(- _wandbruntime‚8542  _runtime‚8542ºÜ³¤à(èO [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16082439747072 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150395071 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 % _timestamp‚1.7884159318695357e+09 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16362285434368 _runtime‚8542.281845364 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87791506 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚321579963538432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚512 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16082608747008 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16036086856114 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 380955399 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87791504 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚321579963538432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚340734020530176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚321948588499758 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _step‚936 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 230560328 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16362070655488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152426145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64634641 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚337662403285504 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150395071 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 230560328 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚321579981541376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚337662403285504 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 380935359 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150395069 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚357096091185664 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚514 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16082608747008 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 230540290 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚337662590288384 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 % _timestamp‚1.7884159318695357e+09 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16362285434368 _runtime‚8542.281845364 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87791506 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚321579963538432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚512 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16082608747008 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16036086856114 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 380955399 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87791504 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚321579963538432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚340734020530176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚321948588499758 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _step‚936 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 230560328 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16362070655488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152426145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64634641 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚337662403285504 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150395071 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408¨§”_U2â(- _wandbruntime‚8542  _runtime‚8542UøÞÃø&ã(òM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚321607232903168 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 230560077 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87798883 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16363513561088 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87798881 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 380987871 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152439190 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _runtime‚8542.936692884 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚337691252804608 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16037476020658 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 230580141 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚305938279704576 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16037641142270 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200085504 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 380987871 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 380967805 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 150407730 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚11399 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚321607250906112 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224025600 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚321975738685874 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16083832885248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _step‚937 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150407728 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16084001898496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150407728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚340762957418496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340762933478400 % _timestamp‚1.7884159325241508e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 230560077 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚337691065788416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚321975920846846 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152439673 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16363713646592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚337691252804608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416c–vä(÷M OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚481 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16083832885248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _step‚937 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150407728 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16084001898496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150407728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚340762957418496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340762933478400 % _timestamp‚1.7884159325241508e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 230560077 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚337691065788416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚321975920846846 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152439673 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16363713646592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚337691252804608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚321607232903168 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 230560077 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejection¦Ýš©‡s‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87798883 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16363513561088 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87798881 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 380987871 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152439190 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _runtime‚8542.936692884 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚337691252804608 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16037476020658 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 230580141 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚305938279704576 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16037641142270 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200085504 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 380987871 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 380967805 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 150407730 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚11399 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚321607250906112 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224025600 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚321975738685874 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 FȧÍî(Ç % _timestamp‚1.7884159670226536e+09 _runtime‚8577.434803396 _step‚939 & lr-AdamW/pg1‚8.626787860597572e-07 & lr-AdamW/pg2‚8.626787860597572e-07 trainer/global_step‚11449«••À 2ï(- _wandbruntime‚8577  _runtime‚8577œn^î'ð(èO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚323315037274202 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚307209478599680 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16152280069632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16152280069632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16105723795926 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16152111069696 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 231553030 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚339095729847808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚342179691881472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚358612884930560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚339095542844928 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 231553030 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16433193049088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16105558674522 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚323315219434966 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _step‚940 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 231532992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚339095542844928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16152111069696 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _runtime‚8577.435856611 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 trainer/global_step‚11449 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153065854 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 231532992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚512 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151028019 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88160354 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151028019 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚339095729847808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚322943449778176 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚510 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚342179667941376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151028021 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚322943449778176 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚322943431775232 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64906010 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚339095542844928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16152111069696 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _runtime‚8577.435856611 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 trainer/global_step‚11449 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153065854 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 231532992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚512 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151028019 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88160354 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151028019 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚339095729847808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚322943449778176 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚510 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚342179667941376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151028021 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚322943449778176 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚322943431775232 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64906010 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚339095729847808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚342179691881472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚358612884930560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚339095542844928 UFD½påƒeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0¬ÚÐ*2ò(- _wandbruntime‚8578  _runtime‚8578ÈÙ~­ø&ó(òM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16107112960678 trainer/global_step‚11449 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚322970701139968 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚358643210638848 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚307235438821376 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16153673221120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 382593457 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 231572843 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚322970719142912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚8578.108159375 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88167731 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚323342551782054 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16106947839066 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚502 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64910858 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚339124392364032 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151040678 % _timestamp‚1.7884159676951704e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _step‚941 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 231572843 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88167733 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚342208604829696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151040680 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚342208580889600 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚358642971933184 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 231552779 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚322970701139968 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 382613523 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16153504207872 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 382593457 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153078589 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153079091 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚500 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151040678 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64911358 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚307235421782016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16153504207872 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 231552779 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚322970701139968 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 382613523 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16153504207872 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 382593457 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153078589 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153079091 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚500 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151040678 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64911358 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚307235421782016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16153504207872 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16153673221120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 382593457 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 231572843 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚322970719142912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚8578.108159375 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88167731 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚323342551782054 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16106947839066 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚502 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64910858 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚339124392364032­!y02õ(- _wandbruntime‚8578  _runtime‚8578¹¯bWö(Ñ 1loss/SRqqt3/detection_loss‚1.2227440083977477 1loss/SRqqt4/detection_loss‚1.9194416327803743 %loss/total_loss‚48.86849303805753 .loss/FRt4/assignment_loss‚3.81161853731895 0loss/FBt2/assignment_loss‚0.4270215921103954 .loss/FRt4/detection_loss‚2.251629333106839 0loss/SRqqt2/detection_loss‚1.819834292680025 /loss/FRt2/detection_loss‚2.2967739371316775 $ _timestamp‚1.788415967697199e+09 .loss/FRt1/assignment_loss‚4.98886722067128 1loss/FBt3/assignment_loss‚0.22358081582315967 _step‚942 2loss/SRqqt1/assignment_loss‚2.2966509265334984 2loss/SRqqt2/assignment_loss‚1.3657024422350028 2loss/SRqqt3/assignment_loss‚1.2396596145398657 0loss/FBt1/assignment_loss‚0.4822533667488137 /loss/FRt1/detection_loss‚3.3089243899221006 0loss/SRqqt1/detection_loss‚2.526809152803923 /loss/FBt2/detection_loss‚3.2233923196792604 epoch‚9 1loss/FBt4/assignment_loss‚0.28758928378468995 .loss/FRt3/detection_loss‚2.058944909196151 /loss/FBt1/detection_loss‚2.5636506974697113 /loss/FBt3/detection_loss‚0.8433539490161956 /loss/FRt2/assignment_loss‚3.480153799057007 /loss/FRt3/assignment_loss‚2.030710971669147 /loss/FBt4/detection_loss‚2.5684283788387594 _runtime‚8578.109257975 trainer/global_step‚11449 2loss/SRqqt4/assignment_loss‚1.6307574665429545?:´\÷(Ö /loss/FRt2/assignment_loss‚3.480153799057007 /loss/FRt3/assignment_loss‚2.030710971669147 /loss/FBt4/detection_loss‚2.5684283788387594 _runtime‚8578.109257975 trainer/global_step‚11449 2loss/SRqqt4/assignment_loss‚1.6307574665429545 1loss/SRqqt3/detection_loss‚1.2227440083977477 1loss/SRqqt4/detection_loss‚1.9194416327803743 %loss/total_loss‚48.86849303805753 .loss/FRt4/assignment_loss‚3.81161853731895 0loss/FBt2/assignment_loss‚0.4270215921103954 .loss/FRt4/detection_loss‚2.251629333106839 0loss/SRqqt2/detection_loss‚1.819834292680025 /loss/FRt2/detection_loss‚2.2967739371316775 $ _timestamp‚1.788415967697199e+09 .loss/FRt1/assignment_loss‚4.98886722067128 1loss/FBt3/assignment_loss‚0.22358081582315967 _step‚942 2loss/SRqqt1/assignment_loss‚2.2966509265334984 2loss/SRqqt2/assignment_loss‚1.3657024422350028 2loss/SRqqt3/assignment_loss‚1.2396596145398657 0loss/FBt1/assignment_loss‚0.4822533667488137 /loss/FRt1/detection_loss‚3.3089243899221006 0loss/SRqqt1/detection_loss‚2.526809152803923 /loss/FBt2/detection_loss‚3.2233923196792604 epoch‚9 1loss/FBt4/assignment_loss‚0.28758928378468995 .loss/FRt3/detection_loss‚2.058944909196151 /loss/FBt1/detection_loss‚2.5636506974697113 /loss/FBt3/detection_loss‚0.8433539490161956®Mã°]‘ø(‚(:† áŸäÔ׆±™ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.936' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1872 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼ˆg!,ù(‚(:¡ áŸäÔð£ñœ disk.rbd1.in‚0 disk.rbd1.out‚ 4.59912109375$ disk./.usageGB‚1098.061206817627( proc.memory.availableMB‚ 250.55859375 disk.sda.in‚0 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.dm-1.in‚0 cpu‚25.379629057489097 disk.rbd2.in‚0 proc.memory.rssMB‚ 2888.53125) proc.memory.percent‚17.63019561767578 network.recv‚ 3.163695e+06 disk.sda.out‚ 339.9921875 disk.sda1.in‚0% disk.nvme0n1.out‚79363.4541015625 disk.dm-0.in‚ 715.55859375" disk.dm-0.out‚79363.4541015625! disk.rbd2.out‚219.33447265625 disk.rbd3.out‚0 disk.nvme0n1.in‚ 715.55859375 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 339.9921875$ memory_percent‚98.47071170806885 proc.cpu.threads‚17 network.sent‚ 2.7796747e+07 disk.sda1.out‚0 disk.dm-1.out‚ 339.9921875 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 715.55859375' disk.nvme0c0n1.out‚79363.4541015625 disk.sda3.in‚0) disk./.usagePercent‚73.69180892425061 disk.drbd1777.out‚0eld‘ú(‚(:† ðŸäÔ²øÆ™ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.039' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8078 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉÑËã.û(‚(:£ ðŸäÔ̓ó› network.recv‚ 3.170432e+06 disk.rbd1.in‚0# disk.dm-0.out‚79363.81884765625 disk.dm-1.in‚0 network.sent‚ 2.7904913e+07 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 715.55859375 disk.rbd2.in‚0 proc.memory.rssMB‚ 2888.53125 disk.sda.in‚0 disk.drbd1777.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 715.55859375 disk.sda3.out‚ 340.2890625 disk.nvme0n1.in‚ 715.55859375! disk.rbd2.out‚220.98681640625 disk.rbd3.out‚0( disk.nvme0c0n1.out‚79363.81884765625( proc.memory.availableMB‚ 250.09765625) proc.memory.percent‚17.63019561767578 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0& disk.nvme0n1.out‚79363.81884765625 disk.sda2.in‚0$ memory_percent‚98.47352504730225 cpu‚25.379032424779375) disk./.usagePercent‚73.69176207487183% disk./.usageGB‚1098.0605087280273 disk.sda.out‚ 340.2890625 disk.rbd1.out‚ 4.662109375 disk.dm-1.out‚ 340.2890625 disk.rbd3.in‚0^UYâ‘ü(‚(:† ÿŸäÔ¬¬£™ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.892' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3784 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¬0ý(‚(:¥ ÿŸäÔðØЛ network.recv‚ 3.174058e+06 disk.nvme0n1.in‚ 715.55859375( disk.nvme0c0n1.out‚79463.71240234375 disk.sda2.out‚0 disk.sda.in‚0) disk./.usagePercent‚73.69176002681156 disk.rbd1.out‚ 4.662109375& disk.nvme0n1.out‚79463.71240234375# disk.dm-0.out‚79463.71240234375 disk.rbd3.in‚0 disk.rbd3.out‚0 proc.memory.rssMB‚ 2888.53125 disk.rbd2.in‚0! disk.rbd2.out‚221.01025390625 disk.sda1.out‚0 disk.dm-0.in‚ 715.55859375 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda.out‚ 340.95703125 disk.sda1.in‚0 cpu‚25.378462476549487 network.sent‚ 2.7910448e+07% disk./.usageGB‚1098.0604782104492$ memory_percent‚98.47502708435059 disk.drbd1777.in‚0 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 340.95703125" disk.nvme0c0n1.in‚ 715.55859375 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 340.95703125' proc.memory.availableMB‚ 249.8515625) proc.memory.percent‚17.63019561767578 proc.cpu.threads‚172óž÷2þ(- _wandbruntime‚8612  _runtime‚8612 ¨±Èÿ( & lr-AdamW/pg2‚7.336585148553974e-07 trainer/global_step‚11499 % _timestamp‚1.7884160024867213e+09 _runtime‚8612.898881429 _step‚943 & lr-AdamW/pg1‚7.336585148553974e-074 ñDÍ€)Ç % _timestamp‚1.7884160024867213e+09 _runtime‚8612.898881429 _step‚943 & lr-AdamW/pg1‚7.336585148553974e-07 & lr-AdamW/pg2‚7.336585148553974e-07 trainer/global_step‚11499¯úª 2)- _wandbruntime‚8612  _runtime‚8612]Ø}R!‚)èO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚324306918014976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16221782392320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16175030492930 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚340528869407232 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚340528682404352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16221782392320 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚360129391323136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153706351 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚343625339292672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 384206703 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65176603 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚343625315352576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 232525694 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚32 @Í4306918014976 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65177145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 232545732 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88529204 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚340528682404352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚324306900012032 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151660969 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16503843543040 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚11499 _step‚944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚324681850370174 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 232545732 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88529206 _runtime‚8612.899935054 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚308506654755840 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16221951392256 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 384186663 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚324306918014976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16221782392320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16175030492930 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚340528869407232 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚340528682404352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16221782392320 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚360129391323136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153706351 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚343625339292672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 384206703 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65176603 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚343625315352576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 232525694 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚324306918014976 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65177145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 232545732 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88529204 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚340528682404352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚324306900012032 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151660969 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16503843543040 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚11499 _step‚944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚324681850370174 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 232545732 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88529206 _runtime‚8612.899935054 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1°¶8´Á2„)- _wandbruntime‚8613  _runtime‚8613‹ÞöÐø&…)òM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚308532580898816 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚324334187379712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚343654252240896 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151673630 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 232565545 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚308532597938176 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16223344543744 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151673628 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚499 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚324709182717262 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 384239175 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16223344543744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 384239175 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚497 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 384219109 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16223175530496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 384219109 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65181937 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16505470395904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 trainer/global_step‚11499 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚340557531923456 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151673628 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16223175530496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88536583 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _step‚945 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 232545481 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65182434 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚340557531923456 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚324709000556290 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 232565545 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚324334187379712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚324334169376768 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153718518 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153719017 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚343654228300800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16176419657474 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 232545481 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚324334169376768 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884160031427357e+09f Pó†)÷M LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 232565545 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚308532597938176 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16223344543744 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151673628 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚499 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚324709182717262 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 384239175 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16223344543744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 384239175 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚497 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 384219109 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16223175530496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 384219109 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65181937 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16505470395904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 trainer/global_step‚11499 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚340557531923456 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151673628 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16223175530496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88536583 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _step‚945 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 232545481 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65182434 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚340557531923456 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚324709000556290 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 232565545 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚324334187379712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚324334169376768 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153718518 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153719017 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚343654228300800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16176419657474 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 232545481 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚324334169376768 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884160031427357e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚308532580898816 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚324334187379712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚343654252240896 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151673630±ÂO-‹2‡)- _wandbruntime‚8613  _runtime‚8613CD÷ ]ˆ)× 0loss/FBt2/assignment_loss‚0.4043611268643872 /loss/FRt1/detection_loss‚3.2114688555399575 /loss/FRt3/detection_loss‚2.1416785616514296 1loss/SRqqt3/detection_loss‚1.3700688448141922 %loss/total_loss‚47.93495691492761 /loss/FRt1/assignment_loss‚4.817603429158528 1loss/SRqqt1/assignment_loss‚2.331125965485206 /loss/FRt2/detection_loss‚2.2106609688864816 1loss/SRqqt4/detection_loss‚2.0980607332153753 /loss/FBt3/detection_loss‚0.8944688364978374 trainer/global_step‚11499 % _timestamp‚1.7884160031446207e+09 _step‚946 /loss/FRt3/assignment_loss‚2.466669337694035 2loss/SRqqt4/assignment_loss‚1.7901980456134134 1loss/SRqqt2/detection_loss‚1.5162365707484158 .loss/FBt2/detection_loss‚2.824224293231964 epoch‚9 0loss/FRt2/assignment_loss‚2.6610796531041463 _runtime‚8613.556655674 /loss/FRt4/detection_loss‚2.1248759445620755 /loss/FBt4/detection_loss‚1.9033270960762387 2loss/SRqqt2/assignment_loss‚1.3929434762179078 2loss/SRqqt3/assignment_loss‚1.1683881889690053 1loss/FBt1/assignment_loss‚0.49762880186608527 0loss/FBt4/assignment_loss‚0.6658751485589337 0loss/SRqqt1/detection_loss‚2.724294355282417 /loss/FBt1/detection_loss‚2.5822691842913628 0loss/FRt4/assignment_loss‚3.8910009599313504 1loss/FBt3/assignment_loss‚0.246448536666862920±?üb‰)Ü /loss/FRt3/assignment_loss‚2.466669337694035 2loss/SRqqt4/assignment_loss‚1.7901980456134134 1loss/SRqqt2/detection_loss‚1.5162365707484158 .loss/FBt2/detection_loss‚2.824224293231964 epoch‚9 0loss/FRt2/assignment_loss‚2.6610796531041463 _runtime‚8613.556655674 /loss/FRt4/detection_loss‚2.1248759445620755 /loss/FBt4/detection_loss‚1.9033270960762387 2loss/SRqqt2/assignment_loss‚1.3929434762179078 2loss/SRqqt3/assignment_loss‚1.1683881889690053 1loss/FBt1/assignment_loss‚0.49762880186608527 0loss/FBt4/assignment_loss‚0.6658751485589337 0loss/SRqqt1/detection_loss‚2.724294355282417 /loss/FBt1/detection_loss‚2.5822691842913628 0loss/FRt4/assignment_loss‚3.8910009599313504 1loss/FBt3/assignment_loss‚0.24644853666686292 0loss/FBt2/assignment_loss‚0.4043611268643872 /loss/FRt1/detection_loss‚3.2114688555399575 /loss/FRt3/detection_loss‚2.1416785616514296 1loss/SRqqt3/detection_loss‚1.3700688448141922 %loss/total_loss‚47.93495691492761 /loss/FRt1/assignment_loss‚4.817603429158528 1loss/SRqqt1/assignment_loss‚2.331125965485206 /loss/FRt2/detection_loss‚2.2106609688864816 1loss/SRqqt4/detection_loss‚2.0980607332153753 /loss/FBt3/detection_loss‚0.8944688364978374 trainer/global_step‚11499 % _timestamp‚1.7884160031446207e+09 _step‚946²ƒJiŠ)‚(:… Ž äÔØ矙 gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚91.903' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.3806 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£ÎA,‹)‚(:¡ Ž äÔÄ”œ disk.rbd2.in‚0 proc.memory.rssMB‚ 2888.53125 disk.sda3.out‚ 341.4140625 disk.sda.in‚0 disk.sda.out‚ 341.4140625& disk.nvme0n1.out‚79562.97802734375 disk.rbd3.in‚0 proc.cpu.threads‚17 network.recv‚ 3.178959e+06) disk./.usagePercent‚73.69178485954238( disk.nvme0c0n1.out‚79562.97802734375# disk.dm-0.out‚79562.97802734375! disk.rbd2.out‚222.66259765625 disk.rbd1.in‚0 disk.dm-0.in‚ 715.55859375 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0$ memory_percent‚98.47369194030762' proc.memory.availableMB‚ 250.0703125) proc.memory.percent‚17.63019561767578 disk.sda1.in‚0 network.sent‚ 2.7916628e+07 disk.rbd3.out‚0 disk.sda2.in‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 715.55859375 disk.dm-1.out‚ 341.4140625 cpu‚25.377918862549908$ disk./.usageGB‚1098.060848236084" disk.nvme0c0n1.in‚ 715.55859375 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1777.out‚0 disk.rbd1.out‚ 4.662109375ÁƒR‘Œ)‚(:†  äÔ«‘³™ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.039' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8078 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆÇé,)‚(:¡  äÔ„ˆéœ disk.rbd1.in‚0& disk.nvme0n1.out‚79662.77587890625$ proc.memory.availableMB‚250.1875 proc.memory.rssMB‚ 2888.53125( disk.nvme0c0n1.out‚79662.77587890625$ disk./.usageGB‚1098.060848236084 disk.dm-1.out‚ 341.98828125! disk.rbd2.out‚224.20166015625 disk.rbd3.out‚0 disk.dm-0.in‚ 715.55859375 disk.dm-1.in‚0 disk.sda2.out‚0 disk.drbd1777.in‚0$ memory_percent‚98.47297668457031 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 341.98828125 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 3.186605e+06 disk.sda2.in‚0 disk.sda.in‚0) proc.memory.percent‚17.63019561767578) disk./.usagePercent‚73.69178485954238 disk.drbd1777.out‚0 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 715.55859375 disk.rbd1.out‚ 4.662109375 disk.sda3.in‚0 disk.nvme0n1.in‚ 715.55859375# disk.dm-0.out‚79662.77587890625 cpu‚25.377346881647867 network.sent‚ 2.8025366e+07 disk.rbd2.in‚0 disk.sda3.out‚ 341.98828125_DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7884160376897004e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚325670386251776 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16291622714880 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 385832355 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚326048481305382 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚341961821963776 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚345070962763776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚309803813872640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚517 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16291622714880 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚515 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326048299144618 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 154347016 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16291453714944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88898054 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚341961821963776 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 385812315 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚361645852067840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚325670368248832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198001664 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152293919 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221941760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16574865363968 trainer/global_step‚11549 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152293921 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 233518396 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 154347533 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 152293921 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65449477 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚309803796833280p¶Mó'“)íO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198001664 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152293919 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221941760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16574865363968 trainer/global_step‚11549 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152293921 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 233518396 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 154347533 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 152293921 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65449477 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚309803796833280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7884160376897004e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚325670386251776 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16291622714880 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 385832355 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚326048481305382 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚341961821963776 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚345070962763776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚309803813872640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚517 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16291622714880 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚515 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326048299144618 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 154347016 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16291453714944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88898054 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚341961821963776 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 385812315 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚361645852067840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚325670368248832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096´[Õÿ 2”)- _wandbruntime‚8648  _runtime‚8648åj< •)ñM K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152306578 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152306580 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 154360259 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65454826 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚309829757054976 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 233558247 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 233538183 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88905431 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152306580 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚341990671482880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 233558247 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚325697655616512 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16293015866368 H:DeviceStatsMonitor.on_train_batch„h%‘ç_end/allocation.all.freed‚ 385844761 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65454306 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚326075813652470 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 385844761 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚341990671482880 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154359737 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16246056597494 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16245891475882 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16576085218816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16292846853120 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚361676195442176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 $ _timestamp‚1.788416038347173e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚361675960930816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16576295790080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚8648.760140586 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16292846853120 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚345099875712000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚325697637613568 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚325697637613568 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 trainer/global_step‚11549 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚341990484466688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 385864827 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚326075631491498 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚325697655616512 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88905433 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 3858648277¸ônü&–)öM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚325697655616512 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88905433 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 385864827 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152306578 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152306580 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 154360259 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65454826 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚309829757054976 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 233558247 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 233538183 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88905431 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152306580 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚341990671482880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 233558247 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚325697655616512 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16293015866368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 385844761 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65454306 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚326075813652470 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 385844761 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚341990671482880 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154359737 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16246056597494 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16245891475882 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16576085218816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16292846853120 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚361676195442176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 $ _timestamp‚1.788416038347173e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚361675960930816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16576295790080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚8648.760140586 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16292846853120 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚345099875712000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚325697637613568 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚325697637613568 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 trainer/global_step‚11549 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚341990484466688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 385864827 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚326075631491498 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360µ/¢fF2—)- _wandbruntime‚8648  _runtime‚8648É5XUW˜)Ñ 1loss/FBt4/assignment_loss‚0.49546349273847834 trainer/global_step‚11549 2loss/SRqqt1/assignment_loss‚2.3899659014752377 /loss/FBt1/assignment_loss‚0.283674437029013 /loss/FRt3/detection_loss‚1.8860281957313418 1loss/SRqqt3/detection_loss‚1.1586209325385946 _runtime‚8648.761563637 _step‚950 1loss/SRqqt3/assignment_loss‚1.364790572651795 %loss/total_loss‚47.74979598917005 1loss/SRqqt2/assignment_loss‚1.286730661355662 2loss/SRqqt4/assignment_loss‚1.5906698390698204 .loss/FRt2/detection_loss‚2.350552785457397 /loss/FBt1/detection_loss‚2.6726032370015194 /loss/FBt3/detection_loss‚0.6819989687370355 .loss/FBt3/assignment_loss‚0.23260089987571 1loss/SRqqt2/detection_loss‚1.7111490274656493 /loss/FBt4/detection_loss‚2.4424917419751484 epoch‚9 0loss/FBt2/assignment_loss‚0.5376703215952148 /loss/FRt4/detection_loss‚2.1034355945057337 1loss/SRqqt1/detection_loss‚2.9701473635175955 % _timestamp‚1.7884160383495142e+09 /loss/FRt1/assignment_loss‚4.921458389308001 0loss/FRt2/assignment_loss‚2.6281361239297047 0loss/FRt3/assignment_loss‚2.4398301442464194 0loss/SRqqt4/detection_loss‚1.902564106365809 /loss/FRt4/assignment_loss‚3.768128311634064 .loss/FRt1/detection_loss‚2.692115600044663 /loss/FBt2/detection_loss‚3.2389693409204483˜«Ë¨\™)Ö 1loss/SRqqt2/assignment_loss‚1.286730661355662 2loss/SRqqt4/assignment_loss‚1.5906698390698204 .loss/FRt2/detection_loss‚2.350552785457397 /loss/FBt1/detection_loss‚2.6726032370015194 /loss/FBt3/detection_loss‚0.6819989687370355 .loss/FBt3/assignment_loss‚0.23260089987571 1loss/SRqqt2/detection_loss‚1.7111490274656493 /loss/FBt4/detection_loss‚2.4424917419751484 epoch‚9 0loss/FBt2/assignment_loss‚0.5376703215952148 /loss/FRt4/detection_loss‚2.1034355945057337 1loss/SRqqt1/detection_loss‚2.9701473635175955 % _timestamp‚1.7884160383495142e+09 /loss/FRt1/assignment_loss‚4.921458389308001 0loss/FRt2/assignment_loss‚2.6281361239297047 0loss/FRt3/assignment_loss‚2.4398301442464194 0loss/SRqqt4/detection_loss‚1.902564106365809 /loss/FRt4/assignment_loss‚3.768128311634064 .loss/FRt1/detection_loss‚2.692115600044663 /loss/FBt2/detection_loss‚3.2389693409204483 1loss/FBt4/assignment_loss‚0.49546349273847834 trainer/global_step‚11549 2loss/SRqqt1/assignment_loss‚2.3899659014752377 /loss/FBt1/assignment_loss‚0.283674437029013 /loss/FRt3/detection_loss‚1.8860281957313418 1loss/SRqqt3/detection_loss‚1.1586209325385946 _runtime‚8648.761563637 _step‚950 1loss/SRqqt3/assignment_loss‚1.364790572651795 %loss/total_loss‚47.74979598917005¶ceèÚ‘š)‚(:† ¬ äÔðÝœ™ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.867' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9734 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û$(›)‚(: ¬ äÔ–î…œ disk.dm-0.in‚ 715.55859375 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 715.55859375 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 715.55859375 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 342.38671875 proc.memory.rssMB‚ 2888.53125 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda1.out‚0 cpu‚25.37678580393052 proc.cpu.threads‚17 disk.rbd0.out‚0! disk.rbd2.out‚224.34228515625& disk.nvme0c0n1.out‚79664.779296875 disk.sda2.out‚0' proc.memory.availableMB‚ 249.7578125 network.recv‚ 3.190484e+06! disk.dm-0.out‚79664.779296875 disk.rbd3.out‚0 disk.sda.out‚ 342.38671875 disk.sda1.in‚0) disk./.usagePercent‚73.66481753790059 disk.drbd1777.in‚0$ disk.nvme0n1.out‚79664.779296875 disk.dm-1.out‚ 342.38671875 disk.rbd0.in‚0 disk.sda.in‚0$ memory_percent‚98.47559928894043 network.sent‚ 2.803073e+07) proc.memory.percent‚17.63019561767578% disk./.usageGB‚1097.6590156555176 disk.sda3.in‚0 disk.rbd1.out‚ 4.662109375Ê"e‘œ)‚(:† » äÔᣙ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.398' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘.ˆ–*)‚(:Ÿ » äÔù³Ðœ network.sent‚ 2.813886e+07 disk.sda3.out‚ 343.19140625" disk.dm-0.out‚79764.1376953125 disk.dm-1.in‚0) proc.memory.percent‚17.63019561767578 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd1777.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0' proc.memory.availableMB‚ 250.0078125% disk./.usageGB‚1097.6589546203613 disk.sda.out‚ 343.19140625! disk.rbd2.out‚225.93994140625% disk.nvme0n1.out‚79764.1376953125 disk.rbd3.out‚0$ memory_percent‚98.47407341003418 cpu‚25.376189888142182 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 715.55859375 disk.rbd1.out‚ 4.662109375 disk.dm-1.out‚ 343.19140625 proc.memory.rssMB‚ 2888.53125 network.recv‚ 3.1971e+06 disk.sda2.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 715.55859375 disk.dm-0.in‚ 715.55859375) disk./.usagePercent‚73.66481344178004' disk.nvme0c0n1.out‚79764.1376953125 proc.cpu.threads‚17àÇV2ž)- _wandbruntime‚8683  _runtime‚8683)òøÄŸ)¾ $ lr-AdamW/pg1‚5.0676726151774e-07 $ lr-AdamW/pg2‚5.0676726151774e-07 trainer/global_step‚11599 % _timestamp‚1.7884160732979496e+09 _runtime‚8683.710172416 _step‚951lT§ôÉ )à $ lr-AdamW/pg2‚5.0676726151774e-07 trainer/global_step‚11599 % _timestamp‚1.7884160732979496e+09 _runtime‚8683.710172416 _step‚951 $ lr-AdamW/pg1‚5.0676726151774e-07·0.ûŸ2¡)- _wandbruntime‚8683  _runtime‚8683Ñc!q$¢)èO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16361294037504 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 234531136 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚327033836485632 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65719203 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚327415112240590 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚327033854488576 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚327033836485632 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚346516610174976 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16361125037568 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 154985582 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16361125037568 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 152926871 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89266906 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 387437967 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16645768957440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16645554178560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89266904 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 R>DeviceStatsMoniºù«H}tor.on_train_batch_start/active_bytes.all.freed‚343394961523200 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 % _timestamp‚1.7884160732986858e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65718678 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚343394961523200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944Žcëó'£)íO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚952 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152926869 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16361125037568 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 152926871 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89266906 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 387437967 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16645768957440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16645554178560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89266904 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚343394961523200 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 % _timestamp‚1.7884160732986858e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65718678 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚343394961523200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16361294037504 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 234531136 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚327033836485632 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65719203 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚327415112240590 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚327033854488576 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚327033836485632 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚346516610174976 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16361125037568 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 154985582 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 387490479 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154998629 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚311126899132416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89274283 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚311126916171776 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚327061123853312 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387490479 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152939528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚327442262426706 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16362687188992 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚343423624026112 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚343423811042304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚327061123853312 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152939530 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16362518175744 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚953 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65724868 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 234550949 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 154999151 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16315363294290 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 234530885 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚363192728586240 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16362518175744 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152939528 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16362687188992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152939530 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16647181522944§9,ý&¦)÷M M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16647181522944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 387490479 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154998629 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚311126899132416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89274283 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚311126916171776 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚327061123853312 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387490479 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152939528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚327442262426706 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16362687188992 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚343423624026112 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚343423811042304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚327061123853312 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152939530 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16362518175744 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚953 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65724868 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 234550949 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 154999151 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16315363294290 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 234530885 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚363192728586240 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16362518175744 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152939528 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16362687188992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152939530¹èóéu2§)- _wandbruntime‚8684  _runtime‚8684;¨£»Z¨)Ô 1loss/SRqqt4/detection_loss‚1.9126742926298403 /loss/FBt1/detection_loss‚2.6884391674628625 $ _timestamp‚1.788416073970545e+09 0loss/FRt2/assignment_loss‚2.9318861314590943 2loss/SRqqt2/assignment_loss‚1.3352340566792658 /loss/FRt1/assignment_loss‚5.204183686164118 0loss/FRt4/assignment_loss‚3.0594069625322637 2loss/SRqqt3/assignment_loss‚1.5020308491702263 1loss/FBt4/assignment_loss‚0.39279960857731144 .loss/FRt4/detection_loss‚2.219296794671279 epoch‚9 _step‚954 0loss/FRt3/assignment_loss‚2.0131664287369206 1loss/FBt1/assignment_loss‚0.44927160238036706 0loss/FBt2/assignment_loss‚0.2134392363805091 /loss/FRt1/detection_loss‚2.8367040657228038 0loss/SRqqt1/detection_loss‚2.584183491509536 1loss/SRqqt3/detection_loss‚1.3361200463886445 _runtime‚8684.382608335 2loss/SRqqt1/assignment_loss‚2.5284086897455413 0loss/FBt3/assignment_loss‚0.2023929030205846 .loss/FBt2/detection_loss‚4.313506841659546 /loss/FBt3/detection_loss‚0.9861936087254933 &loss/total_loss‚48.241780662049834 2loss/SRqqt4/assignment_loss‚1.6593008192584795 .loss/FRt3/detection_loss‚2.109859535154307 trainer/global_step‚11599 .loss/FRt2/detection_loss‚2.133471857360069 0loss/SRqqt2/detection_loss‚1.321055799296924 .loss/FBt4/detection_loss‚2.308754187363845 Æߥ©)Ù 2loss/SRqqt4/assignment_loss‚1.6593008192584795 .loss/FRt3/detection_loss‚2.109859535154307 trainer/global_step‚11599 .loss/FRt2/detection_loss3Ï-º‚2.133471857360069 0loss/SRqqt2/detection_loss‚1.321055799296924 .loss/FBt4/detection_loss‚2.308754187363845 1loss/SRqqt4/detection_loss‚1.9126742926298403 /loss/FBt1/detection_loss‚2.6884391674628625 $ _timestamp‚1.788416073970545e+09 0loss/FRt2/assignment_loss‚2.9318861314590943 2loss/SRqqt2/assignment_loss‚1.3352340566792658 /loss/FRt1/assignment_loss‚5.204183686164118 0loss/FRt4/assignment_loss‚3.0594069625322637 2loss/SRqqt3/assignment_loss‚1.5020308491702263 1loss/FBt4/assignment_loss‚0.39279960857731144 .loss/FRt4/detection_loss‚2.219296794671279 epoch‚9 _step‚954 0loss/FRt3/assignment_loss‚2.0131664287369206 1loss/FBt1/assignment_loss‚0.44927160238036706 0loss/FBt2/assignment_loss‚0.2134392363805091 /loss/FRt1/detection_loss‚2.8367040657228038 0loss/SRqqt1/detection_loss‚2.584183491509536 1loss/SRqqt3/detection_loss‚1.3361200463886445 _runtime‚8684.382608335 2loss/SRqqt1/assignment_loss‚2.5284086897455413 0loss/FBt3/assignment_loss‚0.2023929030205846 .loss/FBt2/detection_loss‚4.313506841659546 /loss/FBt3/detection_loss‚0.9861936087254933 &loss/total_loss‚48.241780662049834ºˆ¿þWœª)‚(:‘ Ê äÔìÓ™ gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.721' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.944200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠÅÜÑ&«)‚(:› Ê äÔÌдœ disk.nvme0n1.in‚ 715.55859375 network.recv‚ 3.202133e+06 disk.rbd2.out‚226.0927734375 disk.rbd3.out‚0 disk.sda.out‚ 343.58984375 disk.drbd1777.in‚0% disk.nvme0n1.out‚79863.2119140625$ memory_percent‚98.47564697265625" proc.memory.availableMB‚249.75) proc.memory.percent‚17.63019561767578 disk.dm-1.in‚0' disk.nvme0c0n1.out‚79863.2119140625 proc.memory.rssMB‚ 2888.53125 disk.rbd1.out‚ 4.662109375 network.sent‚ 2.8145181e+07 disk.rbd2.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda3.in‚0 disk.drbd1777.out‚0" disk.dm-0.out‚79863.2119140625 cpu‚25.375658025097373) disk./.usagePercent‚73.66478912106427 disk.dm-0.in‚ 715.55859375 disk.dm-1.out‚ 343.58984375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.out‚0$ disk./.usageGB‚1097.658592224121 disk.rbd0.out‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 715.55859375 disk.sda3.out‚ 343.58984375 disk.rbd1.in‚0ìWSù¬)‚(:… Ù äÔ·ÎÆ™ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚85.104' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.0208 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Є¹e­)‚(:” Ù äÔÑÕø›) proc.memory.percent‚17.63019561767578 disk.sda.out‚ 344.234375 disk.sda2.out‚0 disk.sda3.out‚ 344.234375 proc.cpu.threads‚17 network.recv‚ 3.20941e+06 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 cpu‚25.375072262294463 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 715.55859375 disk.rbd2.in‚0" disk.nvme0c0n1.in‚ 715.55859375 disk.sda.in‚0 disk.drbd1777.out‚0 disk.dm-1.out‚ 344.234375 disk.rbd3.in‚0( proc.memory.availableMB‚ 249.94921875$ disk./.usageGB‚1097.658592224121 disk.dm-0.out‚79863.76953125$ memory_percent‚98.47443103790283 disk.sda1.out‚0 disk.rbd1.out‚ 4.662109375 disk.dm-1.in‚0 proc.memory.rssMB‚ 2888.53125% disk.nvme0c0n1.out‚79863.76953125 disk.sda2.in‚0 network.sent‚ 2.8253895e+07# disk.nvme0n1.out‚79863.76953125 disk.rbd0.in‚0 disk.rbd2.out‚227.6318359375 disk.rbd3.out‚0) disk./.usagePercent‚73.66478912106427 disk.sda1.in‚0 disk.dm-0.in‚ 715.55859375ØÏ>¦‘®)‚(:† è äÔÜÛûš gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.867' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9734 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!Å©`¯)‚(:“ è äÔ—Т% disk.nvme0c0n1.out‚79962.80078125 disk.dm-0.out‚79962.80078125 disk.rbd2.in‚0 disk.rbd2.out‚227.7724609375 disk.rbd3.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 715.55859375 disk.dm-1.in‚0 disk.rbd0.out‚0 network.sent‚ 2.8259228e+07 disk.sda.in‚0 disk.drbd1777.in‚0 disk.rbd1.out‚ 4.662109375 proc.memory.rssMB‚ 2888.53125) proc.memory.percent‚17.63019561767578 proc.cpu.threads‚17 network.recv‚ 3.213243e+06 disk.sda3.in‚0 disk.sda1.out‚0# disk.nvme0n1.out‚79962.80078125& proc.memory.availableMB‚ 249.953125" disk.nvme0c0n1.in‚ 715.55859375 disk.dm-1.out‚ 344.703125 disk.rbd0.in‚0$ memory_percent‚98.47440719604492) disk./.usagePercent‚73.66478912106427 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 344.703125 disk.rbd1.in‚0 cpu‚25.374506379175674$ disk./.usageGB‚1097.658592224121 disk.sda2.in‚0 disk.drbd1777.out‚0 disk.dm-0.in‚ 715.55859375 disk.sda.out‚ 344.703125(«ÜB2°)- _wandbruntime‚8719  _runtime‚8719º±ööȱ) % _timestamp‚1.7884161086210072e+09 _runtime‚8719.033166496 _step‚955 & lr-AdamW/pg1‚4.089435631815625e-07 & lr-AdamW/pg2‚4.089435631815625e-07 trainer/global_step‚11649`¿åTͲ)Ç _step‚955 & lr-AdamW/pg1‚4.089435631815625e-07 & lr-AdamW/pg2‚4.089435631815625e-07 trainer/global_step‚11649 % _timestamp‚1.7884161086210072e+09 _runtime‚8719.033166496»\ÜE2³)- _wandbruntime‚8719  _runtime‚8719Çß,î'´)èO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153559821 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚347962281526272 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚364678920189440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153559819 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16430965360128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16383611069558 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚328397304722432 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚312398132106240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 389083659 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884161086217003e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚328397322725376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 235503800 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65988964 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 235523838 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89635756 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153559819 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚328781743175798 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚312398115066880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚11649 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16383445948154 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16430965360128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 235523838 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 235503800 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 155625262 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚344828101082624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16716428078592 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚344828101082624 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155624718 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚8719.033968276 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16430796360192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚328397304722432 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚328781561015034 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16430796360192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚364678685664768 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚364678920189440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153559819 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16430965360128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16383611069558 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚328397304722432 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚312398132106240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 389083659 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884161086217003e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚328397322725376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 235503800 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65988964 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 235523838 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89635756 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153559819 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚328781743175798 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚312398115066880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚11649 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16383445948154 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16430965360128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 235523838 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 235503800 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 155625262 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚344828101082624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16716428078592 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚344828101082624 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155624718 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚8719.033968276 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16430796360192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚328397304722432 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚328781561015034 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16430796360192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚364678685664768 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _runtime‚8719.705084885 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153572478 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153572480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚328424574087168 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16717839246336 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚328808893361914 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153572480 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 235523587 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 235543651 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚957 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155637416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚328424592090112 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16432358511616 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389116131 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚¨ŽÜG1006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89643131 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚347991170534400 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚344856950601728 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚364709009780736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 235523587 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244997120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16432189498368 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚364709254777856 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚328809075522886 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚312424058249216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 389096065 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16432189498368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 389096065 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 389116131 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚344856763585536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 235543651 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 trainer/global_step‚11649 % _timestamp‚1.7884161092924418e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16718060303360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚364709254777856 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚328809075522886 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚312424058249216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 389096065 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16432189498368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 389096065 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 389116131 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚344856763585536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 235543651 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 trainer/global_step‚11649 % _timestamp‚1.7884161092924418e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16718060303360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _runtime‚8719.705084885 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153572478 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153572480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚328424574087168 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16717839246336 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚328808893361914 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153572480 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 235523587 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 235543651 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚957 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155637416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚328424592090112 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16432358511616 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389116131 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89643131 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚347991170534400 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚344856950601728 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚364709009780736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 235523587 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0½%écl2¹)- _wandbruntime‚8719  _runtime‚8719VÑL[º)Õ _step‚958 0loss/FRt2/assignment_loss‚2.5097715281972697 0loss/FRt4/assignment_loss‚3.4685981452465056 2loss/SRqqt4/assignment_loss‚1.7632225817390557 2loss/SRqqt2/assignment_loss‚1.1954686546438857 0loss/FBt4/assignment_loss‚0.5457736242031808 _runtime‚8719.706182022 .loss/FBt2/detection_loss‚3.639054766723088 /loss/FBt3/detection_loss‚0.8164387283659859 trainer/global_step‚11649 % _timestamp‚1.7884161092941604e+09 1loss/FBt3/assignment_loss‚0.16674699869933743 /loss/FRt1/assignment_loss‚5.278908530871074 0loss/FRt3/assignment_loss‚2.3193425826910063 0loss/SRqqt4/detection_loss‚1.760107921886033 /loss/FRt4/detection_loss‚1.9707115878661474 1loss/FBt1/assignment_loss‚0.36360599425756845 %loss/total_loss‚48.97638232292938 2loss/SRqqt1/assignment_loss‚2.0955857609709105 /loss/FRt2/detection_loss‚2.2787144651599958 .loss/FBt1/detection_loss‚2.665615938603878 epoch‚9 .loss/FBt4/detection_loss‚2.363432375284342 2loss/SRqqt3/assignment_loss‚1.8744184214196968 0loss/FBt2/assignment_loss‚0.5704773921445719 1loss/SRqqt2/detection_loss‚1.4240608039913298 0loss/SRqqt3/detection_loss‚1.606492891842336 /loss/FRt1/detection_loss‚3.1990938633680344 .loss/FRt3/detection_loss‚2.299311167946676 1loss/SRqqt1/detection_loss‚2.80142759680747988jù`»)Ú 1loss/FBt1/assignment_loss‚0.36360599425756845 %loss/total_loss‚48.97638232292938 2loss/SRqqt1/assignment_loss‚2.0955857609709105 /loss/FRt2/detection_loss‚2.2787144651599958 .loss/FBt1/detection_loss‚2.665615938603878 epoch‚9 .loss/FBt4/detection_loss‚2.363432375284342 2loss/SRqqt3/assignment_loss‚1.8744184214196968 0loss/FBt2/assignment_loss‚0.5704773921445719 1loss/SRqqt2/detection_loss‚1.4240608039913298 0loss/SRqqt3/detection_loss‚1.606492891842336 /loss/FRt1/detection_loss‚3.1990938633680344 .loss/FRt3/detection_loss‚2.299311167946676 1loss/SRqqt1/detection_loss‚2.8014275968074798 _step‚958 0loss/FRt2/assignment_loss‚2.5097715281972697 0loss/FRt4/assignment_loss‚3.4685981452465056 2loss/SRqqt4/assignment_loss‚1.7632225817390557 2loss/SRqqt2/assignment_loss‚1.1954686546438857 0loss/FBt4/assignment_loss‚0.5457736242031808 _runtime‚8719.706182022 .loss/FBt2/detection_loss‚3.639054766723088 /loss/FBt3/detection_loss‚0.8164387283659859 trainer/global_step‚11649 % _timestamp‚1.7884161092941604e+09 1loss/FBt3/assignment_loss‚0.16674699869933743 /loss/FRt1/assignment_loss‚5.278908530871074 0loss/FRt3/assignment_loss‚2.3193425826910063 0loss/SRqqt4/detection_loss‚1.760107921886033 /loss/FRt4/detection_loss‚1.9707115878661474¾ô8Õ¼)‚(:„ ÷ äÔΫ­š gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚122.68' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.536 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡ßÈ.½)‚(:£ ÷ äÔȪ„œ disk.sda.in‚0 disk.dm-0.in‚ 715.55859375 disk.sda3.in‚0$ memory_percent‚98.47619533538818 proc.cpu.threads‚17 network.sent‚ 2.8264686e+07 disk.rbd1.out‚ 4.662109375& disk.nvme0n1.out‚80062.89794921875# disk.dm-0.out‚80062.89794921875 disk.dm-1.in‚0 disk.dm-1.out‚ 345.62109375 disk.rbd2.out‚229.3076171875 disk.sda3.out‚ 345.62109375) disk./.usagePercent‚73.66478912106427 disk.drbd1777.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 cpu‚25.37395777403365 disk.nvme0n1.in‚ 715.55859375 disk.rbd0.in‚0 disk.drbd1777.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 network.recv‚ 3.217011e+06$ disk./.usageGB‚1097.658592224121 disk.sda.out‚ 345.62109375 disk.rbd1.in‚0 proc.memory.rssMB‚ 2888.53125 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 715.55859375( disk.nvme0c0n1.out‚80062.89794921875( proc.memory.availableMB‚ 249.66015625 disk.sda1.out‚0 disk.rbd0.out‚0) proc.memory.percent‚17.63019561767578“öhŒ¾)‚(: †¡äÔ™ãÙ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.5' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìY£E.¿)‚(:£ †¡äÔ³äñ› disk.rbd0.out‚0 disk.rbd2.in‚0! disk.rbd2.out‚229.46044921875 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 4.662109375 proc.memory.rssMB‚ 2888.53125 proc.cpu.threads‚17& disk.nvme0n1.out‚80063.19482421875" disk.nvme0c0n1.in‚ 715.55859375( disk.nvme0c0n1.out‚80063.19482421875 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.drbd1777.out‚0# disk.dm-0.out‚80063.19482421875) disk./.usagePercent‚73.66478912106427$ disk./.usageGB‚1097.658592224121 disk.sda2.out‚0 disk.drbd1777.in‚0( proc.memory.availableMB‚ 250.15234375) proc.memory.percent‚17.63019561767578 disk.rbd1.in‚0 disk.dm-0.in‚ 715.55859375 network.sent‚ 2.837387e+07 disk.sda.out‚ 346.05859375# memory_percent‚98.4731912612915 disk.dm-1.out‚ 346.05859375 disk.rbd0.in‚0 disk.rbd3.in‚0 network.recv‚ 3.225091e+06 disk.sda3.out‚ 346.05859375 disk.sda.in‚0 disk.nvme0n1.in‚ 715.55859375 cpu‚25.373404573283015v t2À)- _wandbruntime‚8754  _runtime‚8754J² &ÉÁ)à ' lr-AdamW/pg1‚3.2156238946929777e-07 ' lr-AdamW/pg2‚3.2156238946929777e-07 trainer/global_step‚11699 $ _timestamp‚1.788416143937636e+09 _runtime‚8754.349748004 _step‚959ÖƸÎÂ)È ' lr-AdamW/pg1‚3.2156238946929777e-07 ' lr-AdamW/pg2‚3.2156238946929777e-07 trainer/global_step‚11699 $ _timestamp‚1.788416143937636e+09 _runtime‚8754.349748004 _step‚959¿~íË2Ã)- _wandbruntime‚8754  _runtime‚8754”ˆ.wî'Ä)èO E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16500467682816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _runtime‚8754.350843187 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 390709311 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16787517178880 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚346261427644928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚329760790962176 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚329760772959232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚346261240642048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 236516540 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16500636682752 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90004606 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 % _timestamp‚1.7884161439384391e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 236516540 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 154192769 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚330148374111006 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16453082887966 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16500467682816 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 156265613 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154192771 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154192769 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚313695291223040 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90004604 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚366195446116352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚349407928937472 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16500636682752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66261009 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚313695274183680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16787308691456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚11699 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚329760772959232 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _step‚960 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 236496502 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚562 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 156266175 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚560 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚346261240642048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 236516540 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16500636682752 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMoÓ/„ÿínitor.on_train_batch_start/inactive_split.large_pool.allocated‚90004606 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 % _timestamp‚1.7884161439384391e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 236516540 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 154192769 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚330148374111006 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16453082887966 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16500467682816 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 156265613 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154192771 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154192769 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚313695291223040 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90004604 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚366195446116352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚349407928937472 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16500636682752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66261009 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚313695274183680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16787308691456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚11699 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚329760772959232 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _step‚960 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 236496502 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚562 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 156266175 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚560 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156278320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚11699 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 236516289 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66266339 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16454306931106 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16501860820992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90011983 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154205428 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16788725743104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚330175524297122 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚329788042323968 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 390721717 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚349436841885696 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚329788060326912 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16502029834240 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 236516289 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚329788042323968 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 390741783 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚313721217366016 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚346290090161152 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚346290090161152 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 156278851 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚330175706458094 _runtime‚8755.026699112 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 390741783 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154205428 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16501860820992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788416144614026e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154205430 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚349436817945600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16454472052718 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 390721717 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚313721234405376 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _step‚961 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 236536353 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16788938411520 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154205428 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16788725743104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚330175524297122 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚329788042323968 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 390721717 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚349436841885696 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚329788060326912 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16502029834240 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 236516289 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚329788042323968 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 390741783 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚313721217366016 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚346290090161152 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚346290090161152 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 156278851 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚330175706458094 _runtime‚8755.026699112 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 390741783 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154205428 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16501860820992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788416144614026e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154205430 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚349436817945600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16454472052718 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 390721717 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚313721234405376 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _step‚961 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 236536353 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16788938411520 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156278320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚11699 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 236516289 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66266339 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16454306931106Á«@«½2É)- _wandbruntime‚8755  _runtime‚8755ƒWÊ)Ñ /loss/FRt2/assignment_loss‚2.670480304834794 /loss/FRt2/detection_loss‚2.2412604768665467 /loss/FRt3/detection_loss‚1.9160451231113935 epoch‚9 $ _timestamp‚1.788416144615933e+09 _step‚962 /loss/FRt1/assignment_loss‚4.755499769002199 1loss/FBt3/assignment_loss‚0.13042661658434931 0loss/SRqqt1/detection_loss‚2.586115255500331 1loss/SRqqt2/assignment_loss‚1.378701940701719 .loss/FRt1/detection_loss‚2.785512225702405 1loss/SRqqt3/detection_loss‚1.5491869744534295 1loss/SRqqt1/assignment_loss‚2.454617603258653 0loss/FBt2/assignment_loss‚0.9325992067654928 .loss/FBt1/detection_loss‚3.007254183292389 /loss/FBt3/detection_loss‚0.8517851776901056 _runtime‚8755.028052211 0loss/FRt3/assignment_loss‚2.4031965803291837 .loss/FRt4/assignment_loss‚4.37997191867163 2loss/SRqqt4/assignment_loss‚1.4205136294834888 /loss/FBt2/detection_loss‚3.5326121846834817 2loss/SRqqt3/assignment_loss‚1.3905979885409276 1loss/FBt1/assignment_loss‚0.42290106608340594 1loss/SRqqt4/detection_loss‚1.6370801100364099 &loss/total_loss‚49.121900818398856 trainer/global_step‚11699 0loss/FBt4/assignment_loss‚0.3882542222768693 .loss/FRt4/detection_loss‚2.165606641492179 0loss/SRqqt2/detection_loss‚1.675882560903566 /loss/FBt4/detection_loss‚2.4457990581339057l`ŒÚ\Ë)Ö 0loss/SRqqt2/detection_loss‚1.675882560903566 /loss/FBt4/detection_loss‚2.4457990581339057 /loss/FRt2/assignment_loss‚2.670480304834794 /loss/FRt2/detection_loss‚2.2412604768665467 /loss/FRt3/detection_loss‚1.9160451231113935 epoch‚9 $ _timestamp‚1.788416144615933e+09 _step‚962 /loss/FRt1/assignment_loss‚4.755499769002199 1loss/FBt3/assignment_loss‚0.13042661658434931 0loss/SRqqt1/detection_loss‚2.586115255500331 1loss/SRqqt2/assignment_loss‚1.378701940701719 .loss/FRt1/detection_loss‚2.785512225702405 1loss/SRqqt3/detection_loss‚1.5491869744534295 1loss/SRqqt1/assignment_loss‚2.454617603258653 0loss/FBt2/assignment_loss‚0.9325992067654928 .loss/FBt1/detection_loss‚3.007254183292389 /loss/FBt3/detection_loss‚0.8517851776901056 _runtime‚8755.028052211 0loss/FRt3/assignment_loss‚2.4031965803291837 .loss/FRt4/assignment_loss‚4.37997191867163 2loss/SRqqt4/assignment_loss‚1.4205136294834888 /loss/FBt2/detection_loss‚3.5326121846834817 2loss/SRqqt3/assignment_loss‚1.3905979885409276 1loss/FBt1/assignment_loss‚0.42290106608340594 1loss/SRqqt4/detection_loss‚1.6370801100364099 &loss/total_loss‚49.121900818398856 trainer/global_step‚11699 0loss/FBt4/assignment_loss‚0.3882542222768693 .loss/FRt4/detection_loss‚2.165606641492179Â?†kì›Ì)‚(: •¡äÔÌû·™ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.751' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚17.750200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹»å!Í)‚(:– •¡äÔàö–œ disk.sda.in‚0 disk.dm-0.in‚ 715.55859375 disk.rbd3.in‚0 cpu‚25.372850590169215 disk.sda.out‚346.625 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 715.55859375( disk.nvme0c0n1.out‚80162.96142578125 disk.sda2.in‚0! disk.rbd2.out‚230.99951171875 disk.rbd3.out‚0 disk.sda2.out‚0 disk.sda3.out‚346.625 disk.dm-1.out‚346.625) proc.memory.percent‚17.63019561767578 network.sent‚ 2.8379952e+07) disk./.usagePercent‚73.66478937707181 disk.rbd1.in‚0 disk.rbd1.out‚ 4.662109375 proc.cpu.threads‚17 network.recv‚ 3.229559e+06% disk./.usageGB‚1097.6585960388184& disk.nvme0n1.out‚80162.96142578125# disk.dm-0.out‚80162.96142578125 disk.rbd0.in‚0$ memory_percent‚98.47779273986816' proc.memory.availableMB‚ 249.3984375 proc.memory.rssMB‚ 2888.53125 disk.sda3.in‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 715.55859375 disk.rbd2.in‚0 disk.drbd1777.out‚0 disk.rbd0.out‚0‹c¯WœÎ)‚(:‘ ¤¡äÔ«¸Ñ™ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.225' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.244999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ .ÇdÏ)‚(:› ¤¡äÔúꛜ disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda.out‚ 347.0390625 d΀éÔÂisk.rbd1.in‚0 disk.dm-0.in‚ 715.55859375 network.sent‚ 2.8488057e+07 proc.memory.rssMB‚ 2888.53125 disk.rbd0.in‚0 disk.nvme0n1.in‚ 715.55859375& disk.nvme0n1.out‚80262.00830078125# disk.dm-0.out‚80262.00830078125) proc.memory.percent‚17.63019561767578 disk.dm-1.out‚ 347.0390625 disk.drbd1777.out‚0 disk.rbd1.out‚ 4.662109375% disk./.usageGB‚1097.6585960388184 disk.rbd2.in‚0( disk.nvme0c0n1.out‚80262.00830078125 disk.sda3.out‚ 347.0390625 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 232.72265625 disk.rbd3.out‚0 disk.sda.in‚0 network.recv‚ 3.236549e+06) disk./.usagePercent‚73.66478937707181" disk.nvme0c0n1.in‚ 715.55859375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.drbd1777.in‚0$ memory_percent‚98.47640991210938# proc.memory.availableMB‚249.625 cpu‚25.372300013701498_TÌ2Ð)- _wandbruntime‚8789  _runtime‚8789ƒ5ÿÆÇÑ)Á _runtime‚8789.943591539 _step‚963 & lr-AdamW/pg1‚2.446419504865814e-07 & lr-AdamW/pg2‚2.446419504865814e-07 trainer/global_step‚11749 $ _timestamp‚1.788416179531418e+09pdéäÌÒ)Æ & lr-AdamW/pg1‚2.446419504865814e-07 & lr-AdamW/pg2‚2.446419504865814e-07 trainer/global_step‚11749 $ _timestamp‚1.788416179531418e+09 _runtime‚8789.943591539 _step‚963Ãq(͆2Ó)- _wandbruntime‚8789  _runtime‚8789ŠDÚî'Ô)èO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚347694567204352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚331514822885450 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 237509242 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90373456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚519 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16570139005440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚314992450339840 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚347694380201472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16858179978752 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16522554706374 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚331124241196032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 237489204 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154825721 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚521 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 156902894 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚347694567204352 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚367711960620544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚331515005046214 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚347694380201472 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 392334963 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66529959 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _step‚964 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚350853576348672 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 237509242 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90373456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚519 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16570139005440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚314992450339840 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚347694380201472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16858179978752 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16522554706374 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚331124241196032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 237489204 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154825721 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚521 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 156902894 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚347694567204352 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚367711960620544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚331515005046214 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚347694380201472 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 392334963 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66529959 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _step‚964 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚350853576348672 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 392367435 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚347723229720576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚500 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚347723229720576 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156915586 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66535255 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚367742295237632 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16859597466624 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154838378 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚11749 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚331151510560768 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154838380 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90380833 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90380831 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154838380 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚331151528563712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 % _timestamp‚1.7884161802145371e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 392347369 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 237529055 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚350882465356800 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 237529055 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 237508991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚367742062823424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16859805940736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚315018376482816 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _runtime‚8790.627506012 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚331542337393302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚350882489296896 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚965 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 156916088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚315018393522176 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 392367435 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚350882465356800 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 237529055 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 237508991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚367742062823424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16859805940736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚315018376482816 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _runtime‚8790.627506012 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚331542337393302 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚350882489296896 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚965 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 156916088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚315018393522176 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 392367435 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 392367435 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚347723229720576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚500 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚347723229720576 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156915586 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66535255 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚367742295237632 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16859597466624 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154838378 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚11749 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚331151510560768 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154838380 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416Åࢋ 2Û)- _wandbruntime‚8790  _runtime‚8790 dZÜ)Ô 1loss/SRqqt1/assignment_loss‚2.329181683751253 % _timestamp‚1.7884161802168174e+09 0loss/FRt3/assignment_loss‚1.7034798069876067 .loss/FRt3/detection_loss‚2.169878071972302 %loss/total_loss‚45.92857109147273 trainer/global_step‚11749 /loss/FRt4/assignment_loss‚2.726921555788621 1loss/FBt3/assignment_loss‚0.20324038098494318 /loss/FBt3/detection_loss‚0.8147767714662365 /loss/FRt1/detection_loss‚3.2368019979733686 1loss/SRqqt1/detection_loss‚2.6384081978064318 epoch‚9 2loss/SRqqt3/assignment_loss‚0.9962502693069046 /loss/FRt2/detection_loss‚2.3291977673121123 1loss/SRqqt2/detection_loss‚1.3576053328056266 0loss/SRqqt3/detection_loss‚1.573383539105239 0loss/SRqqt4/detection_loss‚1.739647314864762 /loss/FBt2/detection_loss‚2.9367918968200684 0loss/FRt1/assignment_loss‚5.2310939477040215 0loss/FRt2/assignment_loss‚2.7704991063346034 2loss/SRqqt2/assignment_loss‚1.3950534002396509 2loss/SRqqt4/assignment_loss‚1.4241195530915747 0loss/FBt4/assignment_loss‚0.4637493320023225 /loss/FBt1/detection_loss‚2.7398673125675748 0loss/FBt1/assignment_loss‚0.3954241285524144 .loss/FRt4/detection_loss‚2.043989893534909 _step‚966 0loss/FBt2/assignment_loss‚0.4805038748309016 .loss/FBt4/detection_loss‚2.228705955669284 _runtime‚8790.628873719ŠïH_Ý)Ù 1loss/SRqqt1/assignment_loss‚2.329181683751253 % _timestamp‚1.7884161802168174e+09 0loss/FRt3/assignment_loss‚1.7034798069876067 .loss/FRt3/detection_loss‚2.169878071972302 %loss/total_loss‚45.92857109147273 trainer/global_step‚11749 /loss/FRt4/assignment_loss‚2.726921555788621 1loss/FBt3/assignment_loss‚0.20324038098494318 /loss/FBt3/detection_loss‚0.8147767714662365 /loss/FRt1/detection_loss‚3.2368019979733686 1loss/SRqqt1/detection_loss‚2.6384081978064318 epoch‚9 2loss/SRqqt3/assignment_loss‚0.9962502693069046 /loss/FRt2/detection_loss‚2.3291977673121123 1loss/SRqqt2/detection_loss‚1.3576053328056266 0loss/SRqqt3/detection_loss‚1.573383539105239 0loss/SRqqt4/detection_loss‚1.739647314864762 /loss/FBt2/detection_loss‚2.9367918968200684 0loss/FRt1/assignment_loss‚5.2310939477040215 0loss/FRt2/assignment_loss‚2.7704991063346034 2loss/SRqqt2/assignment_loss‚1.3950534002396509 2loss/SRqqt4/assignment_loss‚1.4241195530915747 0loss/FBt4/assignment_loss‚0.4637493320023225 /loss/FBt1/detection_loss‚2.7398673125675748 0loss/FBt1/assignment_loss‚0.3954241285524144 .loss/FRt4/detection_loss‚2.043989893534909 _step‚966 0loss/FBt2/assignment_loss‚0.4805038748309016 .loss/FBt4/detection_loss‚2.228705955669284 _runtime‚8790.628873719Æ–‘u‘Þ)‚(:† ¡äÔõ£™ gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.039' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8078 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñè‚7(ß)‚(: ¡äÔ›ê›" proc.memory.rssMB‚ 2798.8203125 network.recv‚ 3.247134e+06$ disk.nvme0n1.out‚80361.552734375 disk.rbd1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.082643508911133% disk./.usageGB‚1097.6590843200684 disk.dm-0.in‚ 715.55859375 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 715.55859375& disk.nvme0c0n1.out‚80361.552734375 disk.sda3.out‚ 348.07421875 disk.sda2.out‚0( disk./.usagePercent‚73.6648221460362 disk.drbd1777.in‚0 disk.dm-1.out‚ 348.07421875 disk.rbd0.out‚0$ memory_percent‚97.92666435241699 disk.sda1.in‚0 disk.sda3.in‚0' proc.memory.availableMB‚ 339.6953125 disk.rbd1.out‚ 4.662109375 disk.sda.out‚ 348.07421875 disk.drbd1777.out‚0! disk.dm-0.out‚80361.552734375 disk.rbd2.in‚0 cpu‚25.371207260658615 disk.nvme0n1.in‚ 715.55859375 disk.sda2.in‚0 proc.cpu.threads‚17 disk.rbd2.out‚ 234.4765625 disk.rbd3.in‚0 network.sent‚ 2.8590098e+07 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0Â+mdœà)‚(:‘ Ñ¡äÔ£¢™ gpu.0.gpu‚23.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.653' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.730600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^ct‚á)‚(:’ Ñ¡äÔ…ñ¬›% disk./.usageGB‚1097.6585693359375 disk.sda.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 proc.cpu.threads‚17 disk.sda3.out‚ 348.765625 disk.rbd1.out‚ 4.662109375 disk.nvme0n1.in‚ 715.55859375 disk.rbd0.out‚0* proc.memory.percent‚17.082643508911133 disk.drbd1777.out‚0 disk.dm-0.in‚ 715.55859375( proc.memory.availableMB‚ 339.63671875 disk.drbd1777.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2798.8203125 network.sent‚ 2.8688199e+07 network.recv‚ 3.253948e+06 disk.sda1.in‚0! disk.dm-0.out‚80460.908203125 disk.dm-1.in‚0 disk.dm-1.out‚ 348.765625 disk.rbd2.out‚234.5 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 715.55859375& disk.nvme0c0n1.out‚80460.908203125 disk.sda2.in‚0$ memory_percent‚97.92702198028564 cpu‚25.37066538233947 disk.sda.out‚ 348.765625$ disk.nvme0n1.out‚80460.908203125 disk.rbd0.in‚0 disk.rbd2.in‚0) disk./.usagePercent‚73.66478758501907˜«»M2â)- _wandbruntime‚8825  _runtime‚8825~F·ÀÊã)Ä ' lr-AdamW/pg1‚1.7819827633761033e-07 ' lr-AdamW/pg2‚1.7819827633761033e-07 trainer/global_step‚11799 % _timestamp‚1.7884162149880168e+09 _runtime‚8825.400161753 _step‚967u¢årÏä)É ' lr-AdamW/pg2‚1.7819827633761033e-07 trainer/global_step‚11799 % _timestamp‚1.7884162149880168e+09 _runtime‚8825.400161753 _step‚967 ' lr-AdamW/pg1‚1.7819827633761033e-07ÇcjÑW2å)- _wandbruntime‚8825  _runtime‚8825ï›ã*î'æ)èO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 238501944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16639810328064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚352299223759872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚332487727435776 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16639979328000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349127519760896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚332487727435776 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 393960615 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16929308963840 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16639979328000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚352299199819776 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16591861403378 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160÷t¿‰ç)íO C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚349127706763776 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 157543828 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90742306 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 238501944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16639810328064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚352299223759872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚332487727435776 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16639979328000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349127519760896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚332487727435776 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 393960615 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16929308963840 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16639979328000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚352299199819776 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16591861403378 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 % _timestamp‚1.7884162156536386e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 393993087 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚504 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 155471330 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16641203466240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16593415689534 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚332908968328510 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚316315535599616 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16641372479488 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 393973021 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚332514996800512 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16641203466240 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155471330 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221928448 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚369258633041408 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚316315552638976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90749681 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 238521757 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚369258854969856 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 238501693 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16641372479488 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16593250567922 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚11799 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 238521757 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚332514996800512 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 393973021 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 238501693 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157556039 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚352328136708096 _step‚969 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155471328 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90749683 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚352328112768000 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349156369280000 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚349156182263808 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 % _timestamp‚1.7884162156536386e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 393993087 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚504 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 155471330 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16641203466240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16593415689534 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚332908968328510 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚316315535599616 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16641372479488 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 393973021 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚332514996800512 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16641203466240 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155471330 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221928448 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚369258633041408 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚316315552638976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90749681 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 238521757 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚369258854969856 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 238501693 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16641372479488 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16593250567922 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚11799 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 238521757 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚332514996800512 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 393973021 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 238501693 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157556039 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚352328136708096 _step‚969 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155471328 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90749683 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚352328112768000 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349156369280000 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚349156182263808 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚350560659320320 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚333851195672576 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17000203653120 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 239474608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67070480 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156091621 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚350560846323200 O@DeviceStatsMonitor.on_trÞ!æ5!ain_batch_start/requested_bytes.all.peak‚ 4757245257 _runtime‚8860.584755406 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 395586267 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚317586751534080 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16709650650624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚353744847230976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156091619 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 158181137 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚499 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚334248084755866 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚370745074824192 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚11849 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333851195672576 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚497 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7884162501723416e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128ŠÜðó'÷)íO O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 _runtime‚8860.584755406 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 395586267 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚317586751534080 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16709650650624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚353744847230976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156091619 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 158181137 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚499 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚334248084755866 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚370745074824192 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚11849 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333851195672576 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚497 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7884162501723416e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67069983 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚972 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 158181636 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91111156 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚350560659320320 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚333851195672576 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17000203653120 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 239474608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67070480 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156091621 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚350560846323200Ìc¬7Ž2ø)- _wandbruntime‚8861  _runtime‚8861`ïTø&ù)òM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17001408913920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156104280 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚333878447034368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91118531 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 239494395 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16711043802112 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16711043802112 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚353773760179200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91118533 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16710874788864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚476 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚474 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16662722386330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 395618739 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 395598673 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156104280 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 158194187 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚350589508839424 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚350589508839424 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚350589321823232 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156104278 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚370775169093120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 % _timestamp‚1.7884162510741374e+09 _step‚973 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333878465037312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67075654 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚317612694716416 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚317612711755776 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158193711 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 395618739 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 239514459 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚333878465037312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚353773784119296 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚334275417102746 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚333878447034368×’…˜‹ú)÷M E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16710874788864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚476 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚474 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16662722386330 UHDeviceStatsMonitor.on_tr¼3brain_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 395618739 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 395598673 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156104280 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 158194187 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚350589508839424 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚350589508839424 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚350589321823232 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156104278 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚370775169093120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 % _timestamp‚1.7884162510741374e+09 _step‚973 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333878465037312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67075654 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚317612694716416 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚317612711755776 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158193711 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 395618739 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 239514459 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚333878465037312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚353773784119296 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚334275417102746 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚333878447034368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17001408913920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156104280 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚333878447034368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91118531 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 239494395 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16711043802112 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16711043802112 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚353773760179200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91118533ÍÛ®Ä2û)- _wandbruntime‚8861  _runtime‚8861΄Yü)Ó /loss/FRt2/assignment_loss‚3.511148237452215 0loss/FRt4/assignment_loss‚3.3907175531574323 1loss/FBt1/assignment_loss‚0.21255287214675758 1loss/SRqqt2/detection_loss‚1.5721345143658774 .loss/FBt4/detection_loss‚2.750012970632977 &loss/total_loss‚46.845688287630956 _runtime‚8861.488248928 2loss/SRqqt3/assignment_loss‚1.0280604463006402 /loss/FBt2/detection_loss‚3.4756192650113786 1loss/FBt4/assignment_loss‚0.46136568616364254 .loss/FBt1/detection_loss‚2.401210806586526 epoch‚9 0loss/FRt3/assignment_loss‚1.9999845828860998 2loss/SRqqt1/assignment_loss‚2.1866420147747827 /loss/FRt3/detection_loss‚1.6999272459652275 /loss/FRt4/detection_loss‚1.9121111754108877 1loss/SRqqt3/detection_loss‚1.2851551985689278 /loss/FBt3/detection_loss‚0.9543328654370274 0loss/SRqqt1/detection_loss‚2.812801296341008 $ _timestamp‚1.788416251076187e+09 .loss/FRt2/detection_loss‚2.233492231490661 /loss/FRt1/assignment_loss‚4.567680568695068 2loss/SRqqt2/assignment_loss‚1.4137066609625306 2loss/SRqqt4/assignment_loss‚1.7038070900873705 ,loss/FRt1/detection_loss‚3.0235759973526 _step‚974 0loss/FBt3/assignment_loss‚0.2016597045605143 0loss/FBt2/assignment_loss‚0.4505967594013782 1loss/SRqqt4/detection_loss‚1.5973925438794223 trainer/global_step‚11849ÁYŽt^ý)Ø trainer/global_step‚11849 /loss/FRt2/assignment_loss‚3.511148237452215 0loss/FRt4/assignment_loss‚3.3907175531574323 1loss/FBt1/assignment_loss‚0.21255287214675758 1loss/SRqqt2/detection_loss‚1.5721345143658774 .loss/FBt4/detection_loss‚2.750012970632977 &loss/total_loss‚46.845688287630956 _runtime‚8861.488248928 2loss/SRqqt3/assignment_loss‚1.0280604463006402 /loss/FBt2/detection_loss‚3.4756192650113786 1loss/FBt4/assignment_loss‚0.46136568616364254 .loss/FBt1/detection_loss‚2.401210806586526 epoch‚9 0loss/FRt3/assignment_loss‚1.9999845828860998 2loss/SRqqt1/assignment_loss‚2.1866420147747827 /loss/FRt3/detection_loss‚1.6999272459652275 /loss/FRt4/detection_loss‚1.9121111754108877 1loss/SRqqt3/detection_loss‚1.2851551985689278 /loss/FBt3/detection_loss‚0.9543328654370274 0loss/SRqqt1/detection_loss‚2.812801296341008 $ _timestamp‚1.788416251076187e+09 .loss/FRt2/detection_loss‚2.233492231490661 /loss/FRt1/assignment_loss‚4.567680568695068 2loss/SRqqt2/assignment_loss‚1.4137066609625306 2loss/SRqqt4/assignment_loss‚1.7038070900873705 ,loss/FRt1/detection_loss‚3.0235759973526 _step‚974 0loss/FBt3/assignment_loss‚0.2016597045605143 0loss/FBt2/assignment_loss‚0.4505967594013782 1loss/SRqqt4/detection_loss‚1.5973925438794223Îg†þ)‚(:… þ¡äÔÂÿˆ™ gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚94.971' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.9942 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­¥Æÿ)‚(: þ¡äԳ歜 disk.drbd1777.out‚0 disk.rbd1.in‚0 disk.rbd2.in‚0! disk.rbd2.out‚237.93017578125& disk.nvme0c0n1.out‚80659.580078125 disk.sda3.in‚0 disk.sda.out‚350.5 cpu‚25.369064400410114 network.sent‚ 2.8808082e+07) disk./.usagePercent‚73.66483622645057 disk.dm-1.out‚350.5 disk.rbd0.out‚0 disk.sda2.out‚0! disk.dm-0.out‚80659.580078125 network.recv‚ 3.268878e+06 disk.dm-0.in‚ 715.55859375 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0$ memory_percent‚97.92764186859131" proc.memory.rssMB‚ 2798.8984375* proc.memory.percent‚17.083120346069336 proc.cpu.threads‚17$ disk./.usageGB‚1097.659294128418" disk.nvme0c0n1.in‚ 715.55859375 disk.sda.in‚0( proc.memory.availableMB‚ 339.53515625 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda3.out‚350.5 disk.drbd1777.in‚0 disk.rbd1.out‚ 4.662109375 disk.nvme0n1.in‚ 715.55859375$ disk.nvme0n1.out‚80659.580078125 disk.rbd3.in‚0 disk.sda1.out‚0Ó“V€*‚(:… ¢äÔ“ÏÌ™ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚95.261' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.0522 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹‹ÿ+*‚(:  ¢äÔ†ƒé› disk.sda.in‚0 disk.drbd1777.out‚0$ memory_percent‚97.92816638946533" disk.nvme0c0n1.in‚ 715.55859375& disk.nvme0c0n1.out‚80759.830078125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 339.44921875 disk.rbd2.in‚0 network.recv‚ 3.27607e+06 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 351.11328125 disk.sda1.out‚0" proc.memory.rssMB‚ 2798.8984375* proc.memory.percent‚17.083120346069336 cpu‚25.36853000023613) disk./.usagePercent‚73.66483494641291 disk.sda2.out‚0 disk.rbd1.out‚ 4.662109375 proc.cpu.threads‚17 disk.sda.out‚ 351.11328125 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 715.55859375$ disk.nvme0n1.out‚80759.830078125 disk.dm-1.out‚ 351.11328125 disk.dm-0.in‚ 715.55859375! disk.dm-0.out‚80759.830078125! disk.rbd2.out‚239.46533203125 network.sent‚ 2.891697e+07% disk./.usageGB‚1097.6592750549316 disk.sda3.in‚0Ä‘•›‚*‚(: œ¢äÔžê«™ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.586' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.717200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“õÜÒ*ƒ*‚(:Ÿ œ¢äÔ¶èû› network.recv‚ 3.280405e+06 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd1.out‚ 4.662109375 disk.rbd0.in‚0 disk.rbd3.out‚0& disk.nvme0c0n1.out‚80858.853515625( proc.memory.availableMB‚ 339.05078125" proc.memory.rssMB‚ 2798.8984375$ disk./.usageGB‚1097.658878326416 disk.sda.in‚0 disk.rbd3.in‚0* proc.memory.percent‚17.083120346069336 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 715.55859375 disk.dm-0.in‚ 715.55859375 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 351.5546875 disk.dm-1.out‚ 351.5546875 disk.rbd0.out‚0$ memory_percent‚97.93059825897217 disk.sda1.out‚0 cpu‚25.367965434639494 proc.cpu.threads‚17) disk./.usagePercent‚73.66480832162935 disk.drbd1777.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 715.55859375 disk.sda.out‚ 351.5546875$ disk.nvme0n1.out‚80858.853515625! disk.dm-0.out‚80858.853515625 disk.dm-1.in‚0! disk.rbd2.out‚239.60595703125 network.sent‚ 2.8922664e+07¶ûð2„*- _wandbruntime‚8896  _runtime‚8896r5{DÈ…* & lr-AdamW/pg1‚7.679442336157044e-08 & lr-AdamW/pg2‚7.679442336157044e-08 trainer/global_step‚11899 % _timestamp‚1.7884162858661766e+09 _runtime‚8896.278370858 _step‚975W-Ô͆*Ç % _timestamp‚1.7884162858661766e+09 _runtime‚8896.278370858 _step‚975 & lr-AdamW/pg1‚7.679442336157044e-08 & lr-AdamW/pg2‚7.679442336157044e-08 trainer/global_step‚11899ÏÂŒº2‡*- _wandbruntime‚8896  _runtime‚8896ÝÇõ¼î'ˆ*èO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156724569 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚335614897851838 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚351993798879744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 240487348 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚318883910650880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16779152973312 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 240467310 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17070924053504 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚318883927690240 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16779321973248 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 158818565 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚355190494642176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91480004 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 397211919 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156724569 trainer/global_step‚11899 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚335214663909376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚513 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 397191879 _runtime‚8896.279344783 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 240487348 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚372261418695680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚335614715691074 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156724571 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚351993985882624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16779152973312 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0•àïÅ&"‰*íO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚335214663909376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚513 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 397191879 _runtime‚8896.279344783 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 240487348 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚372261418695680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚335614715691074 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156724571 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚351993985882624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16779152973312 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156724569 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚335614897851838 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚351993798879744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 240487348 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚318883910650880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16779152973312 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 240467310 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17070924053504 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚318883927690240 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16779321973248 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 158818565 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚355190494642176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMoøRcØÍnitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91480004 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 397211919 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156724569 trainer/global_step‚11899 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0Ð@xTÄ2Š*- _wandbruntime‚8896  _runtime‚8896ʘ¥÷&‹*ñM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91487383 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚335241915271168 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17072336419328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156737230 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16732359326350 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 397224325 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚335642048037954 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16780546111488 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚372291984812544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 240487097 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 158831842 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚355219431530496 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚352022648398848 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚508 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚372291744009728 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _step‚977 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 240507161 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158831332 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67343951 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67344459 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 397244391 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 397244391 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158831332 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67343951 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67344459 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 397244391 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 397244391 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚335241915271168 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17072336419328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156737230 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16732359326350 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 397224325 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚335642048037954 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16780546111488 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚372291984812544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 240487097 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 158831842 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚355219431530496 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚352022648398848 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚508 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚372291744009728 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _step‚977 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 240507161 BÞ2*- _wandbruntime‚8896  _runtime‚8896F\YŽ*Ó 0loss/FRt4/assignment_loss‚3.5424898709020307 1loss/FBt1/assignment_loss‚0.28305123456382053 0loss/FBt3/assignment_loss‚0.1723877861887325 1loss/FBt2/assignment_loss‚0.24914075966392243 .loss/FBt4/detection_loss‚2.249286044921194 2loss/SRqqt2/assignment_loss‚1.4778943932836963 2loss/SRqqt3/assignment_loss‚1.1441663799674384 /loss/FRt2/detection_loss‚2.6032313406467438 /loss/FRt4/detection_loss‚2.1739659568955823 /loss/FRt1/assignment_loss‚5.469365367522607 1loss/SRqqt2/detection_loss‚1.5622745727499325 /loss/FRt2/assignment_loss‚3.340848109938882 /loss/FRt3/assignment_loss‚2.679951866085713 2loss/SRqqt4/assignment_loss‚1.5387770451213185 .loss/FRt3/detection_loss‚1.743711140866463 /loss/FBt3/detection_loss‚0.7294996466519331 _runtime‚8896.936116815 1loss/SRqqt1/assignment_loss‚2.508531943957011 1loss/SRqqt1/detection_loss‚2.9755789836247764 .loss/FBt2/detection_loss‚2.193948813847133 %loss/total_loss‚48.64456532695614 trainer/global_step‚11899 % _timestamp‚1.7884162865240693e+09 0loss/FBt4/assignment_loss‚0.5791372349150541 1loss/SRqqt4/detection_loss‚1.8258232968418222 /loss/FRt1/detection_loss‚3.1105060073045583 0loss/SRqqt3/detection_loss‚1.338054710466947 .loss/FBt1/detection_loss‚3.152942820028825 epoch‚9 _step‚978ö›>^*Ø /loss/FRt1/detection_loss‚3.1105060073045583 0loss/SRqqt3/detection_loss‚1.338054710466947 .loss/FBt1/detection_loss‚3.152942820028825 epoch‚9 _step‚978 0loss/FRt4/assignment_loss‚3.5424898709020307 1loss/FBt1/assignment_loss‚0.28305123456382053 0loss/FBt3/assignment_loss‚0.1723877861887325 1loss/FBt2/assignment_loss‚0.24914075966392243 .loss/FBt4/detection_loss‚2.249286044921194 2loss/SRqqt2/assignment_loss‚1.4778943932836963 2loss/SRqqt3/assignment_loss‚1.1441663799674384 /loss/FRt2/detection_loss‚2.6032313406467438 /loss/FRt4/detection_loss‚2.1739659568955823 /loss/FRt1/assignment_loss‚5.469365367522607 1loss/SRqqt2/detection_loss‚1.5622745727499325 /loss/FRt2/assignment_loss‚3.340848109938882 /loss/FRt3/assignment_loss‚2.679951866085713 2loss/SRqqt4/assignment_loss‚1.5387770451213185 .loss/FRt3/detection_loss‚1.743711140866463 /loss/FBt3/detection_loss‚0.7294996466519331 _runtime‚8896.936116815 1loss/SRqqt1/assignment_loss‚2.508531943957011 1loss/SRqqt1/detection_loss‚2.9755789836247764 .loss/FBt2/detection_loss‚2.193948813847133 %loss/total_loss‚48.64456532695614 trainer/global_step‚11899 % _timestamp‚1.7884162865240693e+09 0loss/FBt4/assignment_loss‚0.5791372349150541 1loss/SRqqt4/detection_loss‚1.8258232968418222ÒîDH9‘*‚(:† «¢äÔØýÉ™ gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.598' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9196 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•p¦,‘*‚(:¡ «¢äÔüöÈ disk.sda3.out‚ 352.3984375 disk.dm-1.in‚0 disk.dm-1.out‚ 352.3984375 disk.rbd0.in‚0 disk.sda1.in‚0" disk.dm-0.out‚80859.8115234375 disk.sda.out‚ 352.3984375% disk./.usageGB‚1097.6600341796875' disk.nvme0c0n1.out‚80859.8115234375 disk.drbd1777.in‚0$ memory_percent‚97.92923927307129) disk./.usagePercent‚73.66488589191222 disk.sda2.in‚0 disk.dm-0.in‚ 715.55859375! disk.rbd2.out‚241.14501953125" proc.memory.rssMB‚ 2798.8984375 disk.rbd0.out‚0 disk.sda.in‚0 disk.rbd1.out‚ 4.662109375' proc.memory.availableMB‚ 339.2734375* proc.memory.percent‚17.083120346069336 network.sent‚ 2.8928094e+07 disk.rbd2.in‚0 cpu‚25.36745012623058 network.recv‚ 3.283855e+06 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 715.55859375 disk.sda3.in‚0 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 715.55859375 disk.sda2.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚80859.8115234375 proc.cpu.threads‚17 disk.rbd3.in‚0 °"œ’*‚(:‘ º¢äÔÅŒ³™ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚101.966' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.393199999999997 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­Flº(“*‚(: º¢äÔƒªªœ( disk./.usagePercent‚73.6648864039273$ disk./.usageGB‚1097.660041809082( disk.nvme0c0n1.out‚80958.94482421875 disk.sda.out‚ 352.78125 disk.drbd1777.out‚0" proc.memory.rssMB‚ 2798.8984375 network.sent‚ 2.9037141e+07 disk.dm-1.in‚0 disk.rbd2.in‚0! disk.rbd2.out‚241.22314453125 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 715.55859375 disk.sda1.out‚0 disk.dm-0.in‚ 715.55859375 network.recv‚ 3.291925e+06 disk.dm-1.out‚ 352.78125 disk.nvme0n1.in‚ 715.55859375& disk.nvme0n1.out‚80958.94482421875 disk.rbd3.in‚0* proc.memory.percent‚17.083120346069336 cpu‚25.36692539045042 proc.cpu.threads‚17 disk.sda1.in‚0# disk.dm-0.out‚80958.94482421875( proc.memory.availableMB‚ 339.01953125 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 352.78125 disk.sda.in‚0$ memory_percent‚97.93078899383545 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.662109375 disk.sda2.out‚0 disk.drbd1777.in‚0ú-:2”*- _wandbruntime‚8931  _runtime‚8931÷ƒ¶ˆÈ•* _step‚979 & lr-AdamW/pg1‚4.185537694548294e-08 & lr-AdamW/pg2‚4.185537694548294e-08 trainer/global_step‚11949 % _timestamp‚1.7884163214208593e+09 _runtime‚8931.832990794·U9SÍ–*Ç & lr-AdamW/pg1‚4.185537694548294e-08 & lr-AdamW/pg2‚4.185537694548294e-08 trainer/global_step‚11949 % _timestamp‚1.7884163214208593e+09 _runtime‚8931.832990794 _step‚979Óù´p2—*- _wandbruntime‚8931  _runtime‚8931ØSH;˜*çO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚499 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67608808 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157357519 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 241480050 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.788416321421568e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚373778183806976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚356636165993472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157357521 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 trainer/global_step‚11949 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 _step‚980 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚336578132146176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16800441980006 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 241480050 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚336578114143232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67609305 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚8931.833908995 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353426938439168 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157357521 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16848824295936 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 241460012 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91848854 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 241460012 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16848993295872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚356636142053376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91848856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157357519 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16848993295872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚353427125442048 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17141800937472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚336578132146176 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 398817531 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚373777942990848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 159457662 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17142017813504 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16800276858602 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 159458161 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353426938439168 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157357521 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16848824295936 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 241460012 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91848854 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 241460012 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16848993295872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚356636142053376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91848856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157357519 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16848993295872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚353427125442048 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17141800937472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚336578132146176 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 398817531 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚373777942990848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 159457662 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17142017813504 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16800276858602 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 159458161 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚8931.833908995 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157370180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚337008861134134 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚353455600942080 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16850217434112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 398870043 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 398849977 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚337008678973162 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11949 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚489 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159470543 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚336605401510912 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚320207029989376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16850386447360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 % _timestamp‚1.7884163220767536e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚353455787958272 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚336605383507968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚336605383507968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159471032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 398870043 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157370178 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚320207012950016 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157370180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚8932.489534435 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 241499863 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157370178 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚981 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 398849977 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67614799 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚356665055001600 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17143211746816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 398870043 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157370178 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚320207012950016 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157370180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚8932.489534435 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 241499863 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157370178 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚981 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 398849977 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67614799 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚356665055001600 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17143211746816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157370180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚337008861134134 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚353455600942080 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16850217434112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 398870043 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 398849977 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚337008678973162 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11949 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚489 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159470543 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚336605401510912 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚320207029989376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16850386447360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 % _timestamp‚1.7884163220767536e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚353455787958272 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚336605383507968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚336605383507968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159471032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16918664618496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92217704 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚375294446474752 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚354860265001472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157990469 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚337941600382976 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚375294678902272 % _timestamp‚1.7884163567645924e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚354860077998592 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157990471 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚358081813404672 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 242452714 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚354860265001472 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 160094426 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67877252 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321478245923840 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 trainer/global_step‚11999 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚354860077998592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚337941582380032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚984 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _runtime‚8967.176886887 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 160094958 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 400443183 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16918664618496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92217704 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚375294446474752 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚354860265001472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157990469 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚337941600382976 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚375294678902272 % _timestamp‚1.7884163567645924e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚354860077998592 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157990471 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚358081813404672 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 242452714 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚354860265001472 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 160094426 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67877252 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321478245923840 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 trainer/global_step‚11999 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚354860077998592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚337941582380032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚984 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _runtime‚8967.176886887 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 160094958 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 400443183 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400495695 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚337968851744768 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16920057769984 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚337968851744768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16919888756736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158003128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚8967.858721206 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 242492565 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16919888756736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92225081 \HDeviceStatsMonitor.on_train_batch_end/inacŸ^JÕtive_split_bytes.all.allocated‚375325010347008 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚338375309908370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 242472501 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17214069228544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17214283994112 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚338375492069342 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚337968869747712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158003130 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160107098 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158003128 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚321504189106176 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚354888927517696 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚354888927517696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 242472501 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 400495695 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 400475629 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67882540 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚375324771641344 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚11999 % _timestamp‚1.7884163574461834e+09 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400495695 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚337968851744768 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16920057769984 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚337968851744768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16919888756736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158003128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚8967.858721206 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 242492565 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16919888756736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92225081 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚375325010347008 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚338375309908370 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 242472501 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17214069228544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17214283994112 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚338375492069342 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚337968869747712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158003130 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160107098 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158003128 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚321504189106176 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚354888927517696 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚354888927517696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 242472501 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 400495695 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 400475629 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67882540 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚375324771641344 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚11999 % _timestamp‚1.7884163574461834e+09 Pa¯*Û 1loss/SRqqt3/detection_loss‚1.4351123864612272 _runtime‚8967.859879248 0loss/FRt2/assignment_loss‚3.1041855241389986 0loss/FRt4/assignment_loss‚3.3176567935943604 1loss/FBt3/assignment_loss‚0.26961030664034297 .loss/FRt4/detection_loss‚2.158872829079628 1loss/SRqqt2/detection_loss‚1.6101798845802324 epoch‚9 % _timestamp‚1.7884163574478197e+09 /loss/FRt3/assignment_loss‚2.407522546052933 /loss/FRt1/detection_loss‚3.1964757960775625 .loss/FBt1/detection_loss‚2.597064795819196 %loss/total_loss‚48.86310260104852 1loss/FBt2/assignment_loss‚0.19507322167945468 /loss/FBt2/detection_loss‚3.7679050139018466 _step‚986 /loss/FRt1/assignment_loss‚5.176875612010127 1loss/FBt1/assignment_loss‚0.41490453722971404 /loss/FRt3/detection_loss‚1.9752813396602869 1loss/SRqqt1/detection_loss‚3.1524911856651308 /loss/FBt4/detection_loss‚2.0785210924223065 1loss/FBt4/assignment_loss‚0.38390835068099705 1loss/SRqqt4/detection_loss‚1.6397719513624907 /loss/FBt3/detection_loss‚0.7406626575224987 1loss/SRqqt1/assignment_loss‚2.389306008219719 0loss/SRqqt2/assignment_loss‚1.40634044184018 2loss/SRqqt4/assignment_loss‚1.6525939929997548 /loss/FRt2/detection_loss‚2.2952280938625336 trainer/global_step‚11999 1loss/SRqqt3/assignment_loss‚1.497558239546995Úؼãþ°*‚(:… ç¢äÔèã¹™ gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.403' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6806 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»1¤Î1±*‚(:¦ ç¢äÔœ¥žœ disk.rbd3.in‚0( disk.nvme0c0n1.out‚81158.24169921875# disk.dm-0.out‚81158.24169921875" proc.memory.rssMB‚ 2798.8984375 cpu‚25.3653403332763$ disk./.usageGB‚1097.660083770752 disk.dm-1.out‚ 354.37109375 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda1.out‚0! disk.rbd2.out‚244.41064453125 disk.sda.out‚ 354.37109375 disk.drbd1777.out‚0$ memory_percent‚97.93274402618408 disk.rbd0.out‚0 disk.rbd1.out‚ 4.662109375 disk.nvme0n1.in‚ 715.55859375 disk.rbd2.in‚0 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 715.55859375 disk.drbd1777.in‚0 disk.rbd1.in‚0 network.recv‚ 3.306711e+06 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-0.in‚ 715.55859375 network.sent‚ 2.9156955e+07 disk.sda3.out‚ 354.37109375 disk.sda1.in‚0& disk.nvme0n1.out‚81158.24169921875( proc.memory.availableMB‚ 338.69921875 proc.cpu.threads‚17) disk./.usagePercent‚73.66488922001018 disk.dm-1.in‚0* proc.memory.percent‚17.083120346069336# ä7²*‚(:… ö¢äÔ‚§ÿš gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚88.957' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.7914 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_6J@1³*‚(:¦ ö¢äÔª•¥ disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 715.55859375 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 338.70703125) disk./.usagePercent‚73.66490330042456 disk.rbd1.out‚ 4.662109375% disk./.usageGB‚1097.6602935791016 disk.dm-0.in‚ 715.55859375# disk.dm-0.out‚81257.51318359375 disk.rbd2.in‚0! disk.rbd2.out‚246.01220703125 disk.sda1.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 715.55859375" proc.memory.rssMB‚ 2798.8984375 disk.dm-1.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚81257.51318359375 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 354.7421875 disk.sda1.out‚0& disk.nvme0n1.out‚81257.51318359375 network.recv‚ 3.314661e+06 disk.dm-1.out‚ 354.7421875 disk.sda2.out‚0 disk.sda3.out‚ 354.7421875$ memory_percent‚97.93269634246826* proc.memory.percent‚17.083120346069336 cpu‚25.364809249511737 disk.drbd1777.out‚0 proc.cpu.threads‚17 network.sent‚ 2.9265934e+07Þ„‡´*‚(:… …£äÔÇʱ™ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.235' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.647 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”)þ/µ*‚(:¤ …£äÔ”ÜÞ› disk.sda3.out‚ 355.68359375 network.recv‚ 3.319143e+06 disk.sda3.in‚0 disk.sda.in‚0$ memory_percent‚97.93291091918945 proc.cpu.threads‚17 cpu‚25.364322698177038) disk./.usagePercent‚73.66489946031155 disk.rbd3.out‚0 disk.sda1.out‚0 disk.drbd1777.out‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 355.68359375 disk.rbd1.in‚0 disk.dm-0.in‚ 715.55859375 disk.rbd0.in‚0! disk.rbd2.out‚246.15283203125 disk.rbd3.in‚0 network.sent‚ 2.9272023e+07 disk.sda1.in‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 715.55859375 disk.dm-1.in‚0 disk.dm-1.out‚ 355.68359375" proc.memory.rssMB‚ 2798.8984375" disk.dm-0.out‚81258.0439453125% disk./.usageGB‚1097.6602363586426' disk.nvme0c0n1.out‚81258.0439453125 disk.rbd1.out‚ 4.662109375% disk.nvme0n1.out‚81258.0439453125& proc.memory.availableMB‚ 338.671875" disk.nvme0c0n1.in‚ 715.55859375 disk.sda2.in‚0* proc.memory.percent‚17.083120346069336ÿ%ñ2¶*- _wandbruntime‚9002  _runtime‚9002­\Ê·*Ä ' lr-AdamW/pg1‚3.5394489642789157e-09 ' lr-AdamW/pg2‚3.5394489642789157e-09 trainer/global_step‚12049 % _timestamp‚1.7884163925339406e+09 _runtime‚9002.946070115 _step‚987×D-<ϸ*É ' lr-AdamW/pg1‚3.5394489642789157e-09 ' lr-AdamW/pg2‚3.5394489642789157e-09 trainer/global_step‚12049 % _timestamp‚1.7884163925339406e+09 _runtime‚9002.946070115 _step‚987Û‰‰J2¹*- _wandbruntime‚9002  _runtime‚9002RS‘º*èO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 243445416 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158623421 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚376810948344832 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218973184 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚356293217558016 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚356293404560896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16988166941184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚322775388001280 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚356293217558016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17283730442240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚339305050616832 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16939220495418 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158623421 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚515 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 243465454 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158623419 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68147846 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚9002.946988495 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚92586556 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158623419 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 243465454 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚322775405040640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 trainer/global_step‚12049 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16988335941120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 402088875 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 402068835 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.sma9úhŸ]ll_pool.peak‚ 239880192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17283511469056 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16988335941120 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 160734402 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68147333GéZˆó'»*íO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚513 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16939385616822 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚376811191258112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚339305050616832 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚356293404560896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚339305068619776 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 % _timestamp‚1.7884163925346725e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚359527436875776 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚339714790657462 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 _step‚988 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚359527460815872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16988335941120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 402088875 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 402068835 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17283511469056 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16988335941120 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 160734402 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68147333 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 243445416 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158623421 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚376810948344832 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218973184 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚356293217558016 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚356293404560896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16988166941184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚322775388001280 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚356293217558016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17283730442240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚339305050616832 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16939220495418 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158623421 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚515 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 243465454 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158623419 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68147846 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚9002.946988495 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚92586556 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158623419 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 243465454 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚322775405040640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 trainer/global_step‚12049 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160746745 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16940774781574 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 trainer/global_step‚12049 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 243465203 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68153327 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚322801348222976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚92593933 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158636080 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚356321880060928 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16989729092608 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158636078 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚356322067077120 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17284930693632 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 243485267 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚339741940843578 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158636080 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚322801331183616 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 $ _timestamp‚1.788416393204754e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92593931 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚339332337984512 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16940609659962 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 402121347 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17285139167744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402121347 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚376841280517632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112b7é·ü&¾*öM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚356321880060928 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16989729092608 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158636078 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚356322067077120 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17284930693632 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 243485267 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚339741940843578 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158636080 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚322801331183616 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 $ _timestamp‚1.788416393204754e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92593931 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚339332337984512 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16940609659962 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 402121347 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17285139167744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402121347 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚376841280517632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 _runtime‚9003.617575505 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158636078 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚339332319981568 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚339332337984512 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚359556349824000 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160746745 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16940774781574 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 trainer/global_step‚12049 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 243465203 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68153327 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚322801348222976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚92593933 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158636080Ý"lÛÍ2¿*- _wandbruntime‚9003  _runtime‚9003::V(À*Ó /loss/FRt4/assignment_loss‚3.195960423402619 1loss/SRqqt3/assignment_loss‚1.202365616550211 /loss/FBt4/detection_loss‚2.9275683675493513 trainer/global_step‚12049 1loss/SRqqt4/assignment_loss‚1.510671352346738 0loss/FBt1/assignment_loss‚0.1478982579177682 1loss/tÑ™1FBt3/assignment_loss‚0.12148954403516274 /loss/FRt4/detection_loss‚1.9700900054814523 0loss/SRqqt3/detection_loss‚1.213893462522108 /loss/FRt1/assignment_loss‚5.140496721634498 1loss/SRqqt4/detection_loss‚1.6637023203728492 % _timestamp‚1.7884163932070181e+09 /loss/FRt2/assignment_loss‚2.825260264308829 epoch‚9 1loss/FBt4/assignment_loss‚0.42579154563177973 /loss/FBt2/detection_loss‚2.9571771383285523 /loss/FBt3/detection_loss‚0.7883907100268769 &loss/total_loss‚47.112359013581994 _runtime‚9003.619107422 _step‚990 2loss/SRqqt2/assignment_loss‚1.4039014467800206 0loss/FBt2/assignment_loss‚0.6592935918757575 .loss/FRt3/detection_loss‚1.952122607723225 0loss/SRqqt1/detection_loss‚2.737195599079132 /loss/FBt1/detection_loss‚3.0347093403339387 0loss/FRt3/assignment_loss‚1.7138314219408257 2loss/SRqqt1/assignment_loss‚2.1742225003242495 /loss/FRt2/detection_loss‚2.4795332033383217 .loss/FRt1/detection_loss‚3.124087778421549 0loss/SRqqt2/detection_loss‚1.742705793656183G‰^Á*Ø .loss/FRt1/detection_loss‚3.124087778421549 0loss/SRqqt2/detection_loss‚1.742705793656183 /loss/FRt4/assignment_loss‚3.195960423402619 1loss/SRqqt3/assignment_loss‚1.202365616550211 /loss/FBt4/detection_loss‚2.9275683675493513 trainer/global_step‚12049 1loss/SRqqt4/assignment_loss‚1.510671352346738 0loss/FBt1/assignment_loss‚0.1478982579177682 1loss/FBt3/assignment_loss‚0.12148954403516274 /loss/FRt4/detection_loss‚1.9700900054814523 0loss/SRqqt3/detection_loss‚1.213893462522108 /loss/FRt1/assignment_loss‚5.140496721634498 1loss/SRqqt4/detection_loss‚1.6637023203728492 % _timestamp‚1.7884163932070181e+09 /loss/FRt2/assignment_loss‚2.825260264308829 epoch‚9 1loss/FBt4/assignment_loss‚0.42579154563177973 /loss/FBt2/detection_loss‚2.9571771383285523 /loss/FBt3/detection_loss‚0.7883907100268769 &loss/total_loss‚47.112359013581994 _runtime‚9003.619107422 _step‚990 2loss/SRqqt2/assignment_loss‚1.4039014467800206 0loss/FBt2/assignment_loss‚0.6592935918757575 .loss/FRt3/detection_loss‚1.952122607723225 0loss/SRqqt1/detection_loss‚2.737195599079132 /loss/FBt1/detection_loss‚3.0347093403339387 0loss/FRt3/assignment_loss‚1.7138314219408257 2loss/SRqqt1/assignment_loss‚2.1742225003242495 /loss/FRt2/detection_loss‚2.4795332033383217ÞDœÂ*‚(:‘ ”£äÔå௙ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.436' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.887200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô&ñ2Ã*‚(:§ ”£äÔØ—Òœ disk.sda3.in‚0 network.sent‚ 2.9277397e+07 network.recv‚ 3.323319e+06( disk./.usagePercent‚73.6651429234766! disk.rbd2.out‚247.68798828125 disk.rbd3.out‚0 disk.sda3.out‚ 356.05859375 disk.sda.in‚0 disk.rbd1.out‚ 4.662109375( disk.nvme0c0n1.out‚81360.95751953125 disk.nvme0n1.in‚ 715.55859375 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda.out‚ 356.05859375 disk.drbd1777.out‚0 disk.dm-0.in‚ 715.55859375 proc.cpu.threads‚17& disk.nvme0n1.out‚81360.95751953125( proc.memory.availableMB‚ 338.63671875 disk.drbd1777.in‚0 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 356.05859375 disk.rbd0.in‚0$ memory_percent‚97.93312549591064" proc.memory.rssMB‚ 2798.8984375* proc.memory.percent‚17.083120346069336% disk./.usageGB‚1097.6638641357422 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.36376826484272 disk.sda2.in‚0# disk.dm-0.out‚81360.95751953125 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 715.55859375»]Ã^Ä*‚(:„ ££äÔÈÁ—™ gpu.0.gpu‚22.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.09' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.218 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž†¬4Å*‚(:© ££äÔÓšžœ proc.cpu.threads‚17% disk./.usageGB‚1097.6638717651367 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd2.in‚0" proc.memory.rssMB‚ 2798.8984375 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.662109375 disk.dm-1.out‚ 356.73046875 disk.rbd3.out‚0 disk.nvme0n1.in‚ 715.55859375& disk.nvme0n1.out‚81461.23095703125! disk.rbd2.out‚247.82861328125 cpu‚25.363250767235492 network.recv‚ 3.329762e+06 disk.sda.out‚ 356.73046875 disk.sda1.in‚0( disk.nvme0c0n1.out‚81461.23095703125# disk.dm-0.out‚81461.23095703125 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 715.55859375* proc.memory.percent‚17.083120346069336 disk.sda2.in‚0 disk.sda1.out‚0$ memory_percent‚97.93198108673096 network.sent‚ 2.9385547e+07 disk.sda.in‚0 disk.dm-0.in‚ 715.55859375 disk.rbd3.in‚0 disk.drbd1777.in‚0 disk.drbd1777.out‚0) disk./.usagePercent‚73.66514343549167 disk.sda3.out‚ 356.73046875( proc.memory.availableMB‚ 338.82421875õ ›Æ*‚(: ²£äÔÙÞŽ™ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.356' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.871199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åf«4Ç*‚(:© ²£äÔ„ïÅœ disk.dm-1.in‚0' proc.memory.availableMB‚ 258.5703125) disk./.usagePercent‚73.66514343549167( disk.nvme0c0n1.out‚81461.48095703125 disk.sda.in‚0 disk.rbd1.out‚ 4.662109375# proc.memory.rssMB‚ 2878.94140625 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 715.55859375 disk.rbd0.out‚0 network.recv‚ 3.334562e+06% disk./.usageGB‚1097.6638717651367 disk.sda2.out‚0 disk.sda3.out‚ 357.02734375 disk.sda.out‚ 357.02734375 disk.drbd1777.in‚0 disk.dm-1.out‚ 357.02734375 cpu‚25.370476604037183 disk.rbd3.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 715.55859375! disk.rbd2.out‚249.36376953125$ memory_percent‚98.42181205749512* proc.memory.percent‚17.571663856506348 network.sent‚ 2.9391959e+07" disk.nvme0c0n1.in‚ 715.55859375 disk.sda1.in‚0 disk.drbd1777.out‚0& disk.nvme0n1.out‚81461.48095703125 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.rbd3.out‚0# disk.dm-0.out‚81461.48095703125d­•È*‚(:… Á£äÔûÌ™ gpu.0.gpu‚42.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚88.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚17.6922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A:¾5É*‚(:ª Á£äÔÆ–Úœ disk.sda1.in‚0! disk.rbd2.out‚249.50439453125 disk.sda.out‚ 357.64453125& disk.nvme0n1.out‚81560.76123046875* proc.memory.percent‚17.571663856506348 network.recv‚ 3.339064e+06) disk./.usagePercent‚73.66514343549167# disk.dm-0.out‚81560.76123046875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚81560.76123046875 disk.sda.in‚0 disk.drbd1777.out‚0 cpu‚25.378752600726973% disk./.usageGB‚1097.6638717651367 disk.rbd2.in‚0 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 715.55859375 disk.rbd3.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 357.64453125 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.662109375 network.sent‚ 2.9398055e+07 disk.dm-1.in‚0 disk.drbd1777.in‚0 disk.nvme0n1.in‚ 715.55859375# proc.memory.rssMB‚ 2878.94140625 disk.dm-0.in‚ 715.55859375 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚98.42102527618408( proc.memory.availableMB‚ 258.69921875 proc.cpu.threads‚17 disk.dm-1.out‚ 357.64453125 disk.rbd0.in‚0º5+ºÊ*‚(:… УäÔÔÏ“™ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.822' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê·r0Ë*‚(:¥ УäÔ€î— disk.rbd1.in‚0# disk.dm-0.out‚81659.84326171875 disk.dm-1.out‚ 358.2578125" disk.nvme0c0n1.in‚ 715.55859375 disk.sda3.out‚ 358.2578125 disk.sda.out‚ 358.2578125 network.sent‚ 2.9403537e+07 disk.sda.in‚0$ memory_percent‚98.42257499694824 proc.cpu.threads‚17) disk./.usagePercent‚73.66514343549167% disk./.usageGB‚1097.6638717651367 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0( disk.nvme0c0n1.out‚81659.84326171875 disk.sda1.out‚0' proc.memory.availableMB‚ 258.4453125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2878.94140625 disk.dm-0.in‚ 715.55859375! disk.rbd2.out‚251.04345703125 disk.rbd3.in‚0 disk.sda3.in‚0 cpu‚25.386285152506044 disk.rbd3.out‚0* proc.memory.percent‚17.571663856506348 disk.drbd1777.out‚0 disk.nvme0n1.in‚ 715.55859375& disk.nvme0n1.out‚81659.84326171875 network.recv‚ 3.34324e+06 disk.rbd1.out‚ 4.810546875 disk.rbd0.out‚0 disk.drbd1777.in‚0[ùk2Ì*- _wandbruntime‚9079  _runtime‚9079þàÅkU*Í*ÏT JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚26 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚17062458499584 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 159667326 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚324716234606592 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 4541568000 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚17013274584906 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚199 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚358409947174400 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚199 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 417333248 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚217 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚17062585807872 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚361673466405888 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 4912020992 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 127308288 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 404291818 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚68611146 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 127308288 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚985 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚217 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚17062458499584 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 5041553408 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚93207513 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 159667326 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚341347506677760 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚358410092485632 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚341729650156450 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚341347506677760 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚274 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚379032501610496 _runtime‚9079.732305417 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 127318528 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚68611420 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 103378432 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 5041553408 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2460 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 159667328 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 145311232 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚358410092485632 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚416 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 269877760 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚361673442465792 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚324716251645952 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚416 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚276 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 140964952 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 159667328 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 4757245257 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚358409947174400 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 _step‚991 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 244640469 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0[™—­Z*Î*ÔT dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚17062585807872 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚361673466405888 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 4912020992 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 127308288 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 404291818 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚68611146 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 127308288 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚985 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚217 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚17062458499584 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 5041553408 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚93207513 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 159667326 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚341347506677760 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚358410092485632 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚341729650156450 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚341347506677760 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚274 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚379032501610496 _runtime‚9079.732305417 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 127318528 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚68611420 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 103378432 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 5041553408 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2460 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 159667328 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 145311232 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚358410092485632 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚416 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 269877760 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚361673442465792 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚324716251645952 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚416 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚276 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 140964952 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 159667328 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 4757245257 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚358409947174400 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 _step‚991 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 244640469 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚26 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚17062458499584 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 159667326 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚324716234606592 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 4541568000 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚17013274584906 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚199 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚358409947174400 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚199 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 417333248 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚217ßÝêk2Ï*- _wandbruntime‚9080  _runtime‚9080»mxÐ*ÙR TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 159678038 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 244647865 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 103378432 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 417333248 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚341747758351498 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 269†Wí]%877760 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2460 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 244647865 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚68616134 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚379052454311936 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7884164702813716e+09 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚361693069971456 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 145311232 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 159678038 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 244631888 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2460 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚985 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚358429306807296 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚17062807635456 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚341747899316450 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 123925592 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚17062807635456 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 404309926 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 417333248 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚416 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 4757245257 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 5041553408 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚416 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚324734153199616 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 161830421 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚93214011 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 410712064 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096‹¾wÓd)Ñ*ÞR hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚361693069971456 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 145311232 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 159678038 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 244631888 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2460 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚985 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚358429306807296 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚17062807635456 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚341747899316450 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 123925592 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚17062807635456 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 404309926 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 417333248 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚416 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 4757245257 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 5041553408 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚416 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚324734153199616 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 161830421 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚93214011 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 410712064 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 159678038 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 244647865 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 103378432 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 417333248 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚341747758351498 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 269877760 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2460 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 244647865 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚68616134 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚379052454311936 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7884164702813716e+09 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2àdƒ½¡Ò*‚(:… ߣäÔö‚˜™ gpu.0.gpu‚43.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚6.508865356445312) gpu.0.memoryAllocatedBytes‚ 5591072768 gpu.0.temp‚30.0 gpu.0.powerWatts‚90.891' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.1782 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î÷7F1Ó*‚(:¦ ߣäÔþ«Öœ disk.sda3.in‚0 disk.nvme0n1.in‚ 715.55859375 disk.dm-1.out‚ 358.6640625 disk.sda2.in‚0# disk.dm-0.out‚81759.19482421875! disk.rbd2.out‚252.69580078125" disk.nvme0c0n1.in‚ 715.55859375# memory_percent‚98.4226942062378* proc.memory.percent‚17.571663856506348 cpu‚25.394539191133248 disk.rbd3.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚81759.19482421875 disk.sda3.out‚ 358.6640625( proc.memory.availableMB‚ 258.42578125 network.sent‚ 2.9409143e+07 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 358.6640625 disk.drbd1777.out‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1777.in‚0& disk.nvme0n1.out‚81759.19482421875 disk.dm-1.in‚0# proc.memory.rssMB‚ 2878.94140625 proc.cpu.threads‚17 network.recv‚ 3.347018e+06) disk./.usagePercent‚73.64769293792773% disk./.usageGB‚1097.4038467407227 disk.rbd1.out‚ 4.810546875 disk.dm-0.in‚ 715.55859375e‚-¨2Ô*- _wandbruntime‚9093  _runtime‚9093ÞÍÈW"+Õ*¾¦ 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4737791121006012 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.02631578966975212 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9855158925056458 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.36666667461395264 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5443704724311829 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9873238205909729 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14002551138401031 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23440398275852203 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.35245537757873535 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9710062146186829 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10694758594036102 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3718954026699066 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9424218535423279 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.537070631980896 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01599462330341339 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18060265481472015 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_4_of_4‚0.002213064581155777 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.30000001192092896 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5147815942764282 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.18505747616291046 +jet/accuracy_2_of_2‚0.10753508657217026 *"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_4‚0.05118440091609955 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23766286671161652 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14339332282543182 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 +particle/specificity‚0.9506141543388367 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5354166626930237 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.26203709840774536 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4482767581939697 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.21837902069091797 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.016592441126704216 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4597285985946655 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9811145067214966 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_7‚0 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9963939189910889 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 (particle/accuracy‚0.8500846028327942 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5092592835426331 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9818713665008545 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184397459030151 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.47226110100746155 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.24542802572250366 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9670138955116272 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1824342906475067 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14176440238952637 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_3‚1 *jet/accuracy_1_of_6‚0.9615384340286255 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.025830520316958427 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_6‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.988676130771637 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 <"Purity/2FRt*SRqqt2FBt/event_purity‚0.0058139534667134285 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.19369369745254517 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5208333134651184 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06792353093624115 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3295454680919647 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13072778284549713 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.059255942702293396 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.00657894741743803 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6795286536216736 particle/accuracy_2_of_5‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 epoch‚9 % _timestamp‚1.7884164827742367e+09 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16943076252937317 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3258100152015686 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9910348653793335 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4517509937286377 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1743447631597519 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6818181872367859 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5333333611488342 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09567900747060776 (particle/f_score‚0.38965505361557007 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.37088117003440857 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12251406162977219 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.15976592898368835 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9912132024765015 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.193452388048172 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 6validation_average_jet_accuracy‚0.4253688156604767 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5059027671813965 )jet/accuracy_1_of_2‚0.739774763584137 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17527340352535248 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1855483204126358 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 particle/accuracy_6_of_7‚1 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.48581862449645996 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqE+ #hqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1493450254201889 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 jet/accuracy_6_of_6‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9847078323364258 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3541887700557709 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0019919080659747124 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.18209879100322723 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5306443572044373 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9952027797698975 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.461612343788147 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9698275327682495 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1493302285671234 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3883838653564453 particle/accuracy_1_of_4‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2569444477558136 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5146304965019226 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.522988498210907 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3235771656036377 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4959861934185028 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,jet/accuracy_4_of_5‚0.004480286967009306 particle/accuracy_2_of_4‚1 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.31666669249534607 jet/accuracy_4_of_6‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3955570161342621 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 particle/accuracy_2_of_2‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2638888955116272 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4037356674671173 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 +jet/accuracy_2_of_3‚0.23284360766410828 ,particle/sensitivity‚0.30672091245651245 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02969783917069435 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5022560954093933 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.25693821907043457 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9923290610313416 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4483124017715454 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5560868382453918 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5404395461082458 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009323306381702423 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1276729255914688 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5274994373321533 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07963967323303223 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4880354702472687 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_6‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0674603208899498 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.022314831614494324 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.015205341391265392 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9871484637260437 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3447916805744171 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.002765753073617816 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365078806877136 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5553537607192993 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009542706422507763 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5514494180679321 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.36240309476852417 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.373727023601532 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3577972650527954 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13769584894180298 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.38966456055641174 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20189549028873444 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.981952428817749 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4778136909008026 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3969298303127289 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18005461990833282 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9870559573173523 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8259800672531128 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.23936781287193298 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.20454546809196472 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.06028369069099426 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.017924528568983078 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 +jet/accuracy_2_of_4‚0.34764450788497925 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.44894763827323914 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151411294937 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5087449550628662 particle/accuracy_4_of_5‚1 +jet/accuracy_3_of_6‚0.09615384787321091 particle/accuracy_4_of_7‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.498100608587265 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.46878674626350403 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04434523731470108 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5175517201423645 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333134651184 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16016055643558502 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4460110068321228 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9851411581039429 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9722574353218079 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13378053903579712 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.375 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1376848816871643 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.03125 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4682593047618866 particle/accuracy_1_of_3‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 *jet/accuracy_1_of_1‚0.4722801148891449 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1789473593235016 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19989177584648132 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3907078504562378 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15311437845230103 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 trainer/global_step‚12089 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20716558396816254 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04301239550113678 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 jet/accuracy_5_of_5‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 _step‚993 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9936589002609253 particle/accuracy_3_of_7‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.48581862449645996 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_5_of_5‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4135802090167999 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14385128021240234 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3352678418159485 particle/accuracy_6_of_6‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5278846025466919 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.024592870846390724 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5364326238632202 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 *jet/accuracy_1_of_4‚0.8948776721954346 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14385128021240234 particle/accuracy_1_of_2‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9886869788169861 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.41452324390411377 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2834460139274597 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5129134058952332 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9823309183120728 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9897217750549316 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5140257477760315 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5789473652839661 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5069444179534912 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.30108869075775146 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5784063339233398 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4844411611557007 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14546187222003937 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4791666567325592 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4548611640930176 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_7‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5447133183479309 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06763818860054016 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 +jet/accuracy_3_of_5‚0.13917051255702972 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1111111044883728 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24131636321544647 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_6‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1295134723186493 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4367827773094177 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.29934656620025635 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 (jet/accuracy_1_of_3‚0.84855055809021 particle/accuracy_3_of_5‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.37059855461120605 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2379215806722641 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13565199077129364 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.49179378151893616 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5298207402229309 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08589304238557816 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9908046126365662 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 +jet/accuracy_2_of_6‚0.49358975887298584 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5376311540603638 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚47.57644619327917 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5942460298538208 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5020889639854431 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005219993181526661 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1975320428609848 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.37268519401550293 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 jet/accuracy_5_of_7‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9898552894592285 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4671158492565155 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4333333373069763 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3728524148464203 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3809601366519928 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.10440612584352493 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10254038870334625 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5184460878372192 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1632198840379715 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1237054318189621 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.286571741104126 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.39669671654701233 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.021995464339852333 particle/accuracy_7_of_7‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4885393977165222 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6818181872367859 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6384162306785583 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.997589111328125 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4642241299152374 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4524385333061218 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5208333134651184 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 _runtime‚9093.188907405 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3315367102622986 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.37200433015823364 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.212808296084404 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03407225012779236 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 +jet/accuracy_3_of_3‚0.02148270234465599 jet/accuracy_2_of_7‚0 jet/accuracy_6_of_7‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 particle/accuracy_5_of_7‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408Š@º•ÏÖ*æ 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9851411581039429 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9722574353218079 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13378053903579712 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.375 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1376848816871643 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.03125 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4682593047618866 particle/accuracy_1_of_3‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 *jet/accuracy_1_of_1‚0.4722801148891449 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1789473593235016 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19989177584648132 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3907078504562378 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15311437845230103 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 trainer/global_step‚12089 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20716558396816254 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04301239550113678 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 jet/accuracy_5_of_5‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 _step‚993 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9936589002609253 particle/accuracy_3_of_7‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.48581862449645996 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_5_of_5‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4135802090167999 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14385128021240234 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3352678418159485 particle/accuracy_6_of_6‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5278846025466919 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.024592870846390724 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5364326238632202 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 *jet/accuracy_1_of_4‚0.8948776721954346 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14385128021240234 particle/accuracy_1_of_2‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9886869788169861 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.41452324390411377 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2834460139274597 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5129134058952332 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9823309183120728 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9897217750549316 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5140257477760315 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5789473652839661 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5069444179534912 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.30108869075775146 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5784063339233398 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4844411611557007 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14546187222003937 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportio\U*{{n‚0.007998512126505375 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4791666567325592 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4548611640930176 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_7‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5447133183479309 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06763818860054016 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 +jet/accuracy_3_of_5‚0.13917051255702972 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1111111044883728 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24131636321544647 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_6‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1295134723186493 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4367827773094177 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.29934656620025635 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 (jet/accuracy_1_of_3‚0.84855055809021 particle/accuracy_3_of_5‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.37059855461120605 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2379215806722641 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13565199077129364 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.49179378151893616 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5298207402229309 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08589304238557816 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9908046126365662 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 +jet/accuracy_2_of_6‚0.49358975887298584 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5376311540603638 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚47.57644619327917 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5942460298538208 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5020889639854431 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005219993181526661 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1975320428609848 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.37268519401550293 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 jet/accuracy_5_of_7‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9898552894592285 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4671158492565155 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4333333373069763 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3728524148464203 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3809601366519928 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.10440612584352493 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10254038870334625 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5184460878372192 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1632198840379715 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1237054318189621 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.286571741104126 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.39669671654701233 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.021995464339852333 particle/accuracy_7_of_7‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4885393977165222 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6818181872367859 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6384162306785583 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.997589111328125 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4642241299152374 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4524385333061218 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5208333134651184 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 _runtime‚9093.188907405 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3315367102622986 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.37200433015823364 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.212808296084404 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03407225012779236 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 +jet/accuracy_3_of_3‚0.02148270234465599 jet/accuracy_2_of_7‚0 jet/accuracy_6_of_7‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 particle/accuracy_5_of_7‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4737791121006012 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.02631578966975212 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9855158925056458 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.36666667461395264 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5443704724311829 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9873238205909729 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14002551138401031 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23440398275852203 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.35245537757873535 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9710062146186829 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10694758594036102 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3718954026699066 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9424218535423279 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.537070631980896 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01599462330341339 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18060265481472015 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_4_of_4‚0.002213064581155777 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.30000001192092896 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5147815942764282 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.18505747616291046 +jet/accuracy_2_of_2‚0.10753508657217026 *"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_4‚0.05118440091609955 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23766286671161652 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14339332282543182 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 +particle/specificity‚0.9506141543388367 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5354166626930237 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.26203709840774536 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4482767581939697 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.21837902069091797 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.016592441126704216 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4597285985946655 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9811145067214966 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_7‚0 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9963939189910889 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 (particle/accuracy‚0.8500846028327942 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5092592835426331 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9818713665008545 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184397459030151 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.47226110100746155 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.24542802572250366 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9670138955116272 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1824342906475067 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14176440238952637 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_3‚1 *jet/accuracy_1_of_6‚0.9615384340286255 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.025830520316958427 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_6‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.988676130771637 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 <"Purity/2FRt*SRqqt2FBt/event_purity‚0.0058139534667134285 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.19369369745254517 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5208333134651184 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06792353093624115 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3295454680919647 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13072778284549713 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.059255942702293396 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.00657894741743803 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6795286536216736 particle/accuracy_2_of_5‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 epoch‚9 % _timestamp‚1.7884164827742367e+09 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16943076252937317 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3258100152015686 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9910348653793335 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4517509937286377 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1743447631597519 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6818181872367859 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5333333611488342 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09567900747060776 (particle/f_score‚0.38965505361557007 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_7‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.37088117003440857 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12251406162977219 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.15976592898368835 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9912132024765015 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.193452388048172 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 6validation_average_jet_accuracy‚0.4253688156604767 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5059027671813965 )jet/accuracy_1_of_2‚0.739774763584137 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17527340352535248 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1855483204126358 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 particle/accuracy_6_of_7‚1 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.48581862449645996 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1493450254201889 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 jet/accuracy_6_of_6‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9847078323364258 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3541887700557709 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0019919080659747124 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.18209879100322723 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5306443572044373 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9952027797698975 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.461612343788147 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9698275327682495 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1493302285671234 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3883838653564453 particle/accuracy_1_of_4‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2569444477558136 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5146304965019226 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.522988498210907 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3235771656036377 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4959861934185028 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,jet/accuracy_4_of_5‚0.004480286967009306 particle/accuracy_2_of_4‚1 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.31666669249534607 jet/accuracy_4_of_6‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3955570161342621 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 particle/accuracy_2_of_2‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2638888955116272 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4037356674671173 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 +jet/accuracy_2_of_3‚0.23284360766410828 ,particle/sensitivity‚0.30672091245651245 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02969783917069435 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5022560954093933 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.25693821907043457 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9923290610313416 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4483124017715454 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5560868382453918 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5404395461082458 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009323306381702423 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1276729255914688 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5274994373321533 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07963967323303223 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4880354702472687 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_6‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0674603208899498 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.022314831614494324 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.015205341391265392 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9871484637260437 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3447916805744171 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.002765753073617816 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365078806877136 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5553537607192993 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009542706422507763 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5514494180679321 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.36240309476852417 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.373727023601532 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3577972650527954 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13769584894180298 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.38966456055641174 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20189549028873444 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.981952428817749 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4778136909008026 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3969298303127289 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18005461990833282 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9870559573173523 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8259800672531128 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.23936781287193298 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.20454546809196472 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.06028369069099426 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.017924528568983078 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 +jet/accuracy_2_of_4‚0.34764450788497925 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.44894763827323914 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151411294937 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5087449550628662 particle/accuracy_4_of_5‚1 +jet/accuracy_3_of_6‚0.09615384787321091 particle/accuracy_4_of_7‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.498100608587265 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.46878674626350403 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04434523731470108 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5175517201423645 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333134651184 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16016055643558502 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4460110068321228 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0á÷”ó¢R×* 1xyn37r0j@ å£äÔÄÌÆ0`Trainer.fit` stopped: `max_epochs=10` reached. ýìFAžØ* 1xyn37r0j‹ æ£äÔØ×ë,zEpoch 9/9 â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â” 1209/1209 0:14:13 • 0:00:00 1.42it/s v_num: 37r0±É”$Ù* 1xyn37r0j æ£äÔ€âÿ, cCX92Ú*- _wandbruntime‚9098  _runtime‚9098@ÓŠÉÛ*‚(’ŠG